Medical Image Processing Apparatus, Luminal Image Processing Apparatus, Luminal Image Processing Method, and Programs for the Same

ABSTRACT

There is provided a medical image processing apparatus including an image-extracting section extracting a frame image from in vivo motion picture data picked up by an in vivo image pickup device or a plurality of consecutively picked-up still image data, and an image analysis section analyzing the frame image extracted by the image-extracting section to output an image analysis result. The image analysis section includes a first biological-feature detection section detecting a first biological feature, a second biological-feature detection section detecting, based on a detection result obtained by the first biological feature detection section, a second biological feature in a frame image picked up temporally before or after the image used for detection by the first biological feature detection section; and a condition determination section making a determination for a biological condition based on a detection result obtained by the second biological feature detection section to output the determination.

TECHNICAL FIELD

The present invention relates to a medical image processing apparatusthat efficiently determines a condition of interest on the basis of alarge amount of biological image data, a luminal image processing and aluminal image processing method which detect the cardia on the basis ofimages of the interior of the lumen, and programs for the apparatusesand method.

BACKGROUND ART

In general, in conventional endoscopic examinations using an endoscope,in vivo image data picked up by an endoscopic apparatus or an endoscopicobservation apparatus are immediately displayed on a display device suchas a CRT and externally stored as motion picture data. A physician viewsthe motion pictures or views frame images in the motion pictures asstill images, during or after examinations for diagnosis.

Further, in recent years, swallowable capsule endoscopes have been used.

For example, as disclosed in Japanese Patent Laid-Open No. 2004-645,image data picked up in vivo with an in-capsule endoscope issequentially externally accumulated as motion picture data by radiocommunication. The physician views the motion pictures or views frameimages in motion pictures as still images for diagnosis.

Furthermore, Japanese Patent Laid-Open No. 2004-188026 discloses anapparatus that applies an image analysis process on still images todisplay the results of the analysis on endoscopic images or in anotherdisplay area.

The image analysis results allow the physician to make diagnosis on thebasis of image analysis values for IHb, vessel analysis, and the like,which are objective determination criteria, without recourse of thephysician's subjective.

However, when the physician views motion pictures after endoscopicexaminations or views motion pictures picked up by an in-capsuleendoscope, the enormous number of frame images contained in the motionpictures results in the need to make much effort in finding a point onthe motion pictures at which a suspected lesion is shown, extractingeach frame image showing the lesion and apply an image analysis processto the image, and making diagnosis on the basis of each image analysisresult.

To solve this problem, a system can be implemented which uses the aboveimage analysis apparatus for still images to apply the same imageanalysis process to all the frame images contained in the motionpictures and to then store the results.

However, the application of the same image analysis process to all theframe images increases processing time, resulting in the need to wait along time until processing results are obtained. Further, a long time isrequired until appropriate processing results are obtained even if theimage analysis process is applied with parameters changed. Anotherdisadvantage is that this method increases the amount of data needing tobe stored until appropriate processing results are obtained.

Furthermore, screening in examinations with an endoscopic apparatusdetermines whether or not the Barrett mucosa or the Barrett esophagus ispresent. The Barrett mucosa is developed when at the junction betweenthe stomach and the esophagus (EG junction), the squamous epitheliumforming the esophagus is replaced with the mucosa of the stomach underthe effect of the reflux esophagitis or the like. The Barrett mucosa isalso called the cylindrical epithelium. If the Barrett mucosa extends atleast 3 cm from the mucosal boundary all along the circumference of across section of the lumen of the esophagus, the patient is diagnosed tohave a disease called the Barrett esophagus.

The incidence of the Barrett esophagus has been increasing particularlyamong Americans and Europeans. The Barrett esophagus is very likely todevelop into the adenocarcinoma and thus has been a major problem.Consequently, it is very important to discover the Barrett mucosa early.

Thus, medical image processing apparatus are desired to objectivelydetermine biological feature values for the Barrett esophagus, theBarrett mucosa, or the like and to provide determinations to theoperator.

Further, as described above, in the medical field, observation anddiagnosis of the organs in the body cavity are widely performed usingmedical equipment having an image pickup function.

For example, in the diagnosis of the esophageal disease, in the case ofthe disease diagnosis of the Barrett esophagus near the EG junction(junction between the stomach and the esophagus) in the upper part ofthe cardia, which corresponds to the boundary between the stomach andthe esophagus, endoscopic examinations are important for the diagnosisof the esophagus because the Barrett esophagus may develop into theadenocarcinoma as described above. An endoscope is inserted into apatient's mouth, and the physician makes the diagnosis of the esophagealdisease while viewing endoscopic images displayed on a monitor screen.

Further, as described above, in recent years, capsule-like endoscopeshave been developed which allow the physician to make the diagnosis ofthe esophageal disease while viewing images obtained with thecapsule-like endoscope. A system has been proposed which detects thedisease on the basis of biological images obtained with a capsule-likeendoscope (see, for example, WO 02/073507 A2).

However, even the above proposed system does not disclose the detectionof the cardia or the vicinity of the cardia boundary based on imagesshowing an area extending from the esophagus to the stomach.

For example, enabling the cardia or the boundary of the cardia to bedetected allows the physician to observe biological tissue images of thedetected cardia or cardia boundary in detail. This enables the diseasesuch as the Barrett esophagus to be quickly diagnosed.

OBJECT OF THE INVENTION

The present invention has been made in view of the above problems. Anobject of the present invention is to provide a medical image processingapparatus that can efficiently determine a condition of interest on thebasis of a large amount of image data.

Another object of the present invention is to provide a luminal imageprocessing apparatus that can detect the cardia on the basis ofintraluminal images.

DISCLOSURE OF INVENTION Means for Solving the Problem

A medical image processing apparatus in accordance with a first aspectof the present invention comprises an image extracting section thatextracts a frame image from in vivo motion picture data picked up by anin vivo image pickup device or a plurality of consecutively picked-upstill image data, and an image analysis section that analyzes the frameimage extracted by the image extracting section to output an imageanalysis result. The image analysis section comprises a first biologicalfeature detection section that detects a first biological feature, asecond biological feature detection section that detects, on the basisof a detection result obtained by the first biological feature detectionsection, a second biological feature in a frame image picked uptemporally before or after the image used for detection by the firstbiological feature detection section, and a condition determinationsection that determines a biological condition on the basis of adetection result obtained by the second biological feature detectionsection to output the determination.

A medical image processing method in accordance with a second aspect ofthe present invention comprises a step of extracting a frame image fromin vivo motion picture data picked up by an in vivo image pickup deviceor a plurality of consecutively picked-up still image data, a step ofanalyzing the extracted frame image to detect a first biologicalfeature, a step of detecting, on the basis of a result of the detectionof the first biological feature, a second biological feature in a frameimage picked up temporally before or after the image used for detectionby the first biological feature detection section, and a step of makinga determination for a biological condition on the basis of a result ofthe detection of the second biological feature to output thedetermination.

A program in accordance with a third aspect of the present inventionallows a computer to execute a function of extracting a frame image fromin vivo motion picture data picked up by an in vivo image pickup deviceor a plurality of consecutively picked-up still image data, a functionof analyzing the extracted frame image to detect a first biologicalfeature, a function of detecting, on the basis of a result of thedetection of the first biological feature, a second biological featurein a frame image picked up temporally before or after the image used fordetection by the first biological feature detection section, and afunction of making a determination for a biological condition on thebasis of a result of the detection of the second biological feature tooutput the determination.

A luminal image processing apparatus in accordance with a fourth aspectof the present invention comprises a feature value calculating sectionthat calculates a predetermined feature value by executing imageprocessing on one or a plurality of intraluminal images obtained bypicking up an image of the gastrointestinal tract and a boundarydetection section that detects a boundary of the gastrointestinal tracton the basis of the calculated feature value.

A luminal image processing method in accordance with the fourth aspectof the present invention comprises a step of calculating a predeterminedfeature value by executing image processing on one or a plurality ofintraluminal images obtained by picking up an image of thegastrointestinal tract and a step of detecting a boundary of thegastrointestinal tract on the basis of the calculated feature value.

A program in accordance with the fourth aspect of the present inventionallows a computer to execute a function of calculating a predeterminedfeature value from one or a plurality of intraluminal images obtained bypicking up an image of the gastrointestinal tract and a function ofdetecting a boundary of the gastrointestinal tract on the basis of thecalculated feature value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram showing the entire configuration of anendoscope system comprising a first embodiment;

FIG. 2 is a diagram schematically showing the parts of the uppergastrointestinal tract endoscopically examined by orally inserting anendoscope;

FIG. 3 is a diagram showing an example of an endoscopic image of thevicinity of the boundary between the esophagus and the stomach;

FIG. 4 is a diagram showing the functional configuration of essentialsections of the image processing apparatus;

FIG. 5 is a diagram showing that motion picture data stored in an imagestorage section is stored as sets of still image data;

FIG. 6A is a diagram showing analysis results stored in an analysisinformation storage section;

FIG. 6B is a diagram showing an example of information used for ananalysis process or set by a processing program storage section 23;

FIG. 7 is a diagram showing an example of a monitor display showing ananalysis result together with an endoscopic image;

FIG. 8 is a flowchart of a process procedure for determining the Barrettesophagus condition;

FIG. 9 is a flowchart showing a process procedure of executing a processof detecting the EG junction, together with information such as imagesused or generated in the procedure;

FIG. 10A is a diagram showing a palisade vessel end point boundary;

FIG. 10B is a diagram showing the palisade vessel end point boundary andan epithelium boundary;

FIG. 10C is a diagram showing that an image of the palisade vessel endpoint boundary or the like is divided by eight radial lines;

FIG. 11 is a flowchart showing the details of a palisade vesselextraction process shown in FIG. 9;

FIG. 12A is a diagram showing an example of an image illustrating anoperation performed for the process shown in FIG. 11;

FIG. 12B is a diagram showing an example of an image illustrating anoperation performed for the process shown in FIG. 11;

FIG. 12C is a diagram showing an example of an image illustrating anoperation performed for the process shown in FIG. 11;

FIG. 13 is a flowchart showing the details of a Barrett mucosadetermination process shown in FIG. 10;

FIG. 14 is a flowchart of a variation of the process shown in FIG. 9;

FIG. 15A is a diagram showing an example of an image illustrating anoperation shown in FIG. 14 and the like;

FIG. 15B is a diagram showing an example of an image illustrating theoperation shown in FIG. 14 and the like;

FIG. 16 is a flowchart showing the details of the Barrett mucosadetermination process shown in FIG. 14;

FIG. 17 is a diagram showing the functional configuration of essentialsections of an image processing apparatus in accordance with a secondembodiment;

FIG. 18 is a flowchart of a process procedure of determining the Barrettesophagus condition in accordance with the second embodiment;

FIG. 19 is a flowchart showing a process procedure of executing a cardiadetection process, together with information such as images which isused or generated in the procedure;

FIG. 20A is a diagram illustrating an operation shown in FIG. 19;

FIG. 20B is a diagram illustrating an operation shown in FIG. 19;

FIG. 21 is a flowchart showing the details of a concentration levelcalculation process shown in FIG. 19;

FIG. 22 is a flowchart showing a closed cardia determination processshown in FIG. 19;

FIG. 23 is a flowchart showing a process procedure of executing a cardiadetection process in accordance with a variation, together withinformation such as images which is used or generated in the procedure;

FIG. 24A is a diagram illustrating an operation shown in FIGS. 23 and25;

FIG. 24B is a diagram illustrating the operation shown in FIGS. 23 and25;

FIG. 24C is a diagram illustrating the operation shown in FIGS. 23 and25;

FIG. 25 is a flowchart showing the details of an edge componentgeneration angle calculation process shown in FIG. 23;

FIG. 26 is a flowchart showing an open cardia determination processshown in FIG. 23;

FIG. 27 is a diagram showing the functional configuration of essentialsections of an image processing apparatus in accordance with Example 2;

FIG. 28 is a flowchart of a process procedure of determining the Barrettesophagus condition;

FIG. 29 is a flowchart of a process procedure of determining the Barrettesophagus condition in accordance with a variation;

FIG. 30A is a block diagram showing the general configuration of acapsule endoscope apparatus in accordance with a fourth embodiment;

FIG. 30B is a block diagram showing the general configuration of aterminal apparatus serving as a luminal image processing apparatus inaccordance with the fourth embodiment;

FIG. 31 is a diagram illustrating the general structure of the capsuleendoscope in accordance with the fourth embodiment;

FIG. 32 is a flowchart showing an example of the flow of a process ofdetecting the cardia by passing through the EG junction, the processbeing executed by the terminal apparatus;

FIG. 33 is a schematic graph illustrating a variation in the color tonein a series of endoscopic images obtained;

FIG. 34 is a flowchart showing an example of the flow of a process instep S203 shown in FIG. 32;

FIG. 35 is a flowchart showing an example of the flow of a process ofdetecting a variation in average color tone feature value by calculatinga differential value for average color tone feature, values;

FIG. 36 is a graph illustrating a variation in the standard deviation orvariance of the color tone feature in a series of endoscopic imagesobtained in accordance with a seventh variation of the fourthembodiment;

FIG. 37 is a diagram showing an example of areas of a frame image whichare subjected to image processing in accordance with the fourthembodiment and a variation thereof;

FIG. 38 is a schematic graph illustrating a variation in the brightnessof a series of endoscopic images obtained, specifically, a variation inluminance, in accordance with a fifth embodiment;

FIG. 39 is a flowchart showing an example of the flow of a process ofdetecting the cardia upon passage through the EG junction, the processbeing executed by a terminal apparatus on the basis of the series ofendoscopic images obtained in accordance with the fifth embodiment;

FIG. 40 is a flowchart showing an example of the flow of a process instep S33 shown in FIG. 39;

FIG. 41 is a schematic graph illustrating a variation in G or B pixeldata in the series of endoscopic images, the G or B pixel data beingused as brightness information on the images instead of the luminancecalculated from the three pixel values for R, G, and B as describedabove;

FIG. 42 is a flowchart showing an example of the flow of a process ofdetecting a variation in brightness by calculating a differential valuefor average luminance values in accordance with the fifth embodiment;

FIG. 43 is a diagram showing an example of an image in which a capsuleendoscope is located in front of the open cardia in accordance with asixth embodiment;

FIG. 44 is a flowchart showing an example of the flow of a process ofdetecting the open cardia on the basis of a series of endoscopic imagesin accordance with the sixth embodiment;

FIG. 45 is a diagram showing an example of an image obtained when thecapsule endoscope passes through the open cardia in accordance with aseventh embodiment;

FIG. 46 is a flowchart showing an example of a process of detecting theopen cardia on the basis of a series of endoscopic images in accordancewith the seventh embodiment;

FIG. 47 is a diagram showing a filter property observed during abandpass filtering process in accordance with the seventh embodiment;

FIG. 48 is a diagram showing an example of an image resulting from theprocess of predetermined bandpass filtering and binarization executed onthe image shown in FIG. 45;

FIG. 49 is a flowchart showing an example of the flow of a process ofdetecting the cardia on the basis of a series of endoscopic imagesobtained in accordance with an eighth embodiment;

FIG. 50 is a diagram showing an image of an extracted boundary inaccordance with the eighth embodiment;

FIG. 51 is a diagram showing an example of an image resulting from theprocess of predetermined bandpass filtering and binarization executed ona processing target image in accordance with the eighth embodiment;

FIG. 52 is a flowchart showing an example of the flow of a process ofdetecting the cardia on the basis of a series of endoscopic imagesobtained in accordance with a ninth embodiment;

FIG. 53 is a diagram showing the position of a centroid calculated by adark area centroid coordinate calculation process in accordance with theninth embodiment;

FIG. 54 is a diagram illustrating the evaluation of a circumferentialcharacter in accordance with the ninth embodiment;

FIG. 55 is a diagram illustrating that the evaluation of thecircumferential character is based on area rate in accordance with afourth variation of the ninth embodiment;

FIG. 56 is a diagram illustrating that the evaluation of thecircumferential character is based on angular range in accordance with afourth variation of a tenth embodiment;

FIG. 57 is a diagram showing an example of an image in which the capsuleendoscope is located in front of the closed cardia in accordance withthe tenth embodiment;

FIG. 58 is a flowchart showing an example of the flow of a process ofdetecting the cardia on the basis of a series of endoscopic imagesobtained in accordance with the tenth embodiment;

FIG. 59 is a flowchart showing an example of the flow of a process ofdetecting the cardia on the basis of a series of endoscopic imagesobtained in accordance with an eleventh embodiment;

FIG. 60 is a diagram showing an example of an image illustrating thecardia shape expressed with thin lines on the basis of the image of theclosed cardia, in accordance with the eleventh embodiment;

FIG. 61 is a flowchart showing an example of the flow of a process ofcalculating the variance value, corresponding to the concentration levelparameter, in accordance with the eleventh embodiment; and

FIG. 62 is a diagram showing an example of an image illustratingbranching points in accordance with the eleventh embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described with reference tothe drawings.

First Embodiment

FIGS. 1 to 16 relate to a first embodiment. FIG. 1 shows the entireconfiguration of an endoscopic system comprising the present embodiment.FIG. 2 schematically shows the parts of the upper gastrointestinal tractendoscopically examined by orally inserting an endoscope. FIG. 3 showsan example of an endoscopic image of the vicinity of the boundarybetween the esophagus and the stomach. FIG. 4 shows the functionalconfiguration of an image processing apparatus in accordance with thepresent embodiment. FIG. 5 shows that motion picture data stored in animage storage section is stored as sets of still image data.

FIGS. 6A and 6B show analysis results stored in an analysis informationstorage section, information stored in a processing program storagesection, and the like. FIG. 7 shows an example of a monitor displayshowing an analysis result together with an endoscopic image. FIG. 8 isa flowchart of a process procedure for determining the Barrett esophaguscondition in accordance with the present embodiment. FIG. 9 shows aprocess procedure of executing a process of detecting the EG junction,together with information such as images used or generated.

FIGS. 10A to 10C are diagrams showing the boundary of an end point ofthe palisade vessel. FIG. 11 is a flowchart of a palisade vesselextraction process shown in FIG. 9. FIGS. 12A to 12C show an example ofan image illustrating an operation performed for the process shown inFIG. 11. FIG. 13 is a flowchart of a Barrett mucosa determinationprocess shown in FIG. 10. FIG. 14 is a flowchart of a variation of theprocess shown in FIG. 9. FIGS. 15A and 15B show an example of an imageillustrating an operation shown in FIG. 14 and the like. FIG. 16 is aflowchart of the Barrett mucosa determination process in FIG. 14.

An endoscopic system 1 shown in FIG. 1 is composed of an endoscopicobservation apparatus 2, a medical image processing apparatus(hereinafter simply referred to as an image processing apparatus) 3composed of a personal computer or the like to execute image processingon images obtained by the endoscopic observation apparatus 2, and amonitor 4 that displays the images subjected to the image processing bythe image processing apparatus 3.

The endoscopic observation apparatus 2 has an endoscope 6 forming an invivo image pickup device inserted into the lumen to pick up images ofthe interior of the body, a light source device 7 that suppliesillumination light to the endoscope 6, a camera control unit(hereinafter simply referred to as a CCU) 8 that executes signalprocessing for the image pickup means of the endoscope 6, and a monitor9 to which video signals outputted by the CCU 8 are inputted to displayendoscopic images picked up by an image pickup device.

The endoscope 6 has an insertion portion 11 inserted in the body cavityand an operation portion 12 provided at a trailing end of the insertionportion 11. Further, a light guide 13 is placed inside the insertionportion 11 to transmit illumination light.

A trailing end of the light guide 13 is connected to the light sourcedevice 7. Illumination light supplied by the light source device 7 istransmitted by the light guide 13. The (transmitted) illumination lightis then emitted from a distal plane attached to an illumination windowprovided at a distal end 14 of the insertion portion 11 to illuminate asubject such as a diseased part.

An image pickup apparatus 17 is provided which comprises an objectivelens 15 attached to an observation window located adjacent to theillumination window and for example, a charge coupled device(hereinafter referred to as a CCD) 16 located at a position where theobjective lens 15 forms an image and serving as a solid-state imagepickup device. An optical image formed on an image pickup surface of theCCD 16 is photoelectrically converted by the CCD 16.

The CCD 16 is connected to the CCU 8 via a signal line to output thephotoelectrically converted image signal in response to the applicationof a CCD driving signal from the CCU 8. The image signal is subjected tosignal processing by a video processing circuit in the CCU 8 and thusconverted into a video signal. The video signal is outputted to themonitor 9, which thus displays the endoscopic image on a display surfacethereof. The video signal is also inputted to the image processingapparatus 3.

In the present embodiment, the endoscope 6 is used in the followingcase. The distal end 14 of the insertion portion 11 of the endoscope 6is inserted through the mouth of the patient down to the vicinity of theboundary between the esophagus and the stomach to determine whether ornot the Barrett mucosa is present near the boundary; the Barrett mucosais the normal mucosa (specifically, the squamous epithelium) of theesophagus, the mucosa to be detected, modified to exhibit the conditionof the mucosa part of the stomach.

In this case, a video signal corresponding to an endoscopic imageobtained by picking up an image of the surface of the biological mucosain the body is also inputted to the image processing apparatus 3. Animage processing method described below is executed on the video signalto detect (determine) whether or not the Barrett mucosa is present orthe state of a disease called the Barrett esophagus has been reached.

The image processing apparatus 3 has an image input section 21 to whicha video signal corresponding to the endoscopic image inputted by theendoscopic observation apparatus 2 is inputted, a CPU 22 serving as acentral processing unit to execute image processing on image datainputted by the image input section 21, and a processing program storagesection 23 that stores a processing program (control program) thatallows the CPU 22 to execute image processing.

Further, the image processing apparatus 3 has an image storage section24 that stores image data and the like inputted by the image inputsection 21, an analysis information storage section 25 that storesanalysis information and the like processed by the CPU 22, a hard disk27 serving as a storage device that stores the image data, analysisinformation, and the like processed by the CPU 22, via a storage deviceinterface 26, a display processing section 28 that executes a displayprocess for displaying the image data and the like processed by the CPU22, and an input operation section 29 comprising a keyboard and the likeand used by the user to input data such as image processing parametersand to perform instruction operations.

The video signal generated by the display processing section 28 isoutputted to the display monitor 4 to display the processed imagesubjected to image processing, on the display surface of the displaymonitor 4. The image input section 21, the CPU 22, the processingprogram storage section 23, the image storage section 24, the analysisinformation storage section 25, the storage device interface 26, thedisplay processing section 28, and the input operation section 29 areconnected together via a data bus 30.

In the present embodiment, an examination or diagnosis target site isthe circumferential portion of the junction between the esophagus andthe stomach. An image obtained by the endoscope 6 is subjected to imageanalysis to determine whether or not a suspected site of the Barrettesophagus is present, that is, to make a condition determination.

Thus, the insertion portion 11 of the endoscope 6 is inserted into thepatient's mouth from the distal end of the insertion portion 11 toperform image pickup. FIG. 2 is a figure showing a luminal site in whichthe distal end of the endoscope is positioned when the endoscope 6 isorally inserted into the body cavity of the patient. The distal end 14of the endoscope 6 is inserted into the mouth 31 and advances from theesophagus inlet 32 into the esophagus 33. The distal end 14 of theendoscope 6 moves through the epithelium boundary 34 and the EG junction35 to the stomach 36 and then via the cardia 37 to the interior of thestomach 36.

The operation of inserting the endoscope 6 allows the acquisition ofmotion picture data picked up in the above order. The motion picturedata thus acquired is stored in the image storage section 24. Imageanalysis is executed on frame images of still images constituting themotion picture data.

FIG. 3 is a schematic diagram of an example of a picked-up endoscopicimage of the vicinity of the boundary between the esophagus 33 and thestomach 36. In the endoscopic image, the cardia 37 is an inlet to theinterior of the stomach and is opened and closed.

The palisade vessels 38 substantially radially running outside thecardia 37 are present only in the esophagus 33 side. The palisadevessels 38 extend in the vertical direction along the lumen of theesophagus 33.

Further, an area extending from the epithelium boundary 34 (shown by analternate long and short dash line) corresponding to the boundarybetween the mucosal tissue in the esophagus 33 side and the mucosaltissue in the stomach 36 side to the cardia has a very reddish mucosalcolor tone (the epithelium in which this color tone is distributed iscalled the columnar epithelium). An area extending in the oppositedirection has a whitish mucosal color tone (the epithelium in which thiscolor tone is distributed is called the squamous epithelium). Thisenables the epithelium boundary to be determined by endoscopicobservations.

A line (shown by a dashed line) joining the end points of the palisadevessels 38 together is a boundary line (in fact, the line is notpresent) that cannot be easily identified by endoscopic observations.The line is called the EG junction 35 and corresponds to the tissueboundary between the stomach 36 and the esophagus 33.

The epithelium boundary 34 is normally located near the EG junction 35.However, if the reflux esophagitis or the like replaces the squamousepithelium forming the esophagus 33 with the mucosa (columnar epitheliumor Barrett mucosa) of the stomach 36, the epithelium boundary 39 risestoward the esophagus 33.

If the Barrett mucosa is formed at least 3 cm away from the normalmucosal boundary all along the circumference of the cross section of theesophagus lumen, the patient is diagnosed to have the Barrett esophagus.

FIG. 4 shows the functional configuration of essential sections of theimage processing apparatus 3.

Image data on motion pictures with its image picked up by the endoscope6 and inputted to the image processing apparatus 3 is stored, as motionpicture data Vm1, Vm2, . . . , in the image storage section 24, servingas image storage (image recording) means.

In this case, the motion picture data Vm1, Vm2, . . . have a datastructure in which still images are accumulated over time. Thus, whenthe motion picture data Vm1, Vm2, . . . are stored in the image storagesection 24, for example, as shown in FIG. 5, frame numbers 0, 1, . . . ,MAX_COUNT are assigned to the still image data, which are thus labeledas Vs0, Vs1, . . . , VsM (M=MAX_COUNT).

Further, frame time simultaneously stored in the image storage section24 is stored. The still image data may be compressed in accordance withJPEG or the like before being stored.

When image processing is started, the CPU 22 and processing programallow an image extracting block 41 composed of software to extract andread the still image data within the range indicated by specified framenumbers from, for example, the motion picture data Vm1 read from theimage storage section 24. The image extracting block 41 constitutes animage extracting section that extracts frame image data from in vivomotion picture data or data on a plurality of consecutively picked-upstill images.

Extracted still image data are sequentially sent to an image analysisblock 42 and a display processing block 43.

The image analysis block 42 comprises an epithelium boundary detectionblock 44 that detects epithelium boundary, an EG junction detectionblock 45 that detects the EG junction, and a Barrett esophagusdetermination block 46 that determines whether or not the patient hasthe Barrett esophagus. The image analysis block 42 constitutes an imageanalysis section that analyzes the frame image extracted by the imageextracting block 41 to output an image analysis result.

The epithelium boundary detection block 44, for example, detects avariation in mucosa color tone in an image as an edge to detect anepithelium boundary line present in the image as a point sequence.

The EG junction detection block 45, for example, detects a line joiningthe end points of the palisade vessels together as a point sequence (amethod for detection will be described below in detail).

The Barrett esophagus determination block 46 calculates feature valuessuch as the shape of the epithelium boundary, the striped residue of thesquamous epithelium, the distance between the epithelium boundary andthe EG junction, the standard deviation of the distance, and the maximumand minimum values of the distance to determine whether or not thetarget site with its image picked up indicates the Barrett esophagus.

Information on the determination made by the Barrett esophagusdetermination block 46 is stored in the analysis information storagesection 25, and sent to the display processing block 43. The informationon the determination based on the analysis executed by the imageanalysis block 42 is displayed in a still image shown on the monitor 4via the image extracting block 41.

FIG. 6A shows an example of analysis results stored in the analysisinformation storage section 25. FIG. 6B shows an example of informationused or set when the processing program storage section 23 executes ananalysis process.

Further, FIG. 7 shows a display example in which information on adetermination is displayed in an analyzed still image on the monitor 4.

As described with reference to FIG. 8, to make a condition determinationof whether or not any still image data in the motion picture datacontains the Barrett esophagus, the present embodiment determineswhether or not an image of a reference site (in the present embodiment,the EG junction) comprising a first biological feature (value) waspicked up temporally before or after (substantially simultaneously with)the pickup of an image of a determination target site to be subjected toa Barrett esophagus condition determination. The EG junction detectionblock 45 constitutes a first biological feature detection section thatdetects the first biological feature.

The present embodiment is characterized by executing such an imageprocessing procedure as described below if the determination processdetermines that an image of the reference site has been picked up. Asecond biological feature (value) (in the present embodiment, a featureof the epithelium boundary) is detected in a still image in a framefollowing or preceding the frame of the reference site. Then, on thebasis of the detection result of the second biological feature, aBarrett esophagus determination is made. This allows an efficientdetermination to be made for the Barrett esophagus condition, thecondition determination target. The epithelium boundary detection block44 constitutes a second biological feature detection section thatdetects the second biological feature in the frame image picked uptemporally before or after the image used for the detection by the EGjunction detection block 45, on the basis of the detection result fromthe EG junction detection block 45.

Such image analysis processing makes it possible to omit, for example, aprocess of detecting the second biological feature in images notcomprising the first biological feature. This allows a conditiondetermination to be efficiently made for the condition determinationtarget in a short time. A large amount of image data can thus beappropriately processed.

Now, with reference to the flowchart in FIG. 8, description will begiven of the operation of the image processing apparatus 3 in accordancewith the present embodiment.

When the user uses the input operation section 29 to specify a file namefor motion picture data to the CPU 22, which executes a process inaccordance with a processing program, the CPU 22 reads the maximumnumber of frames for the specified motion picture data, from the imagestorage section 24. As shown in FIG. 6B, the maximum frame number issubstituted into a parameter MAX-COUNT indicating the maximum framenumber to start a process in accordance with the processing program.

In the first step S1, the CPU 22 initializes a frame number variableCOUNT, that is, sets COUNT=0.

In the next step S2, the CPU 22 compares the frame number variable COUNTwith MAX_COUNT. If COUNT>MAX_COUNT, the process is ended.

If step S2 results in the opposite determination, that is,COUNT≦MAX_COUNT, the process proceeds to step S3 where the imageextracting block 41 extracts an image with a frame number=COUNT.

In the next step S4, the EG junction detection block 45 executes, inaccordance with the present embodiment, a process of detecting the EGjunction in the image with that frame number as a process of detectingthe first biological feature (the biological feature is hereinaftersimply referred to as the feature).

Depending on whether or not the detection result indicates a pointsequence of a line indicating the EG junction 35, the CPU 22 determineswhether or not the EG junction 35 is present as shown in step S5.

If the CPU 22 determines in step S5 that the EG junction 35 is notpresent, the CPU 22 suspends the process in steps S3 and S4 to proceedto the next step S6. The CPU 22 then increments the frame numbervariable COUNT by one and returns to step S2 to repeat the process insteps S2 to S6.

On the other hand, in step S5, if the CPU 22 determines that the EGjunction 35 is present, the CPU 22 detects the second feature in stepS7, and on the basis of the detection result, shifts to a conditiondetermination process of determining whether or not the patient has theBarrett esophagus, the condition determination target.

In step S7, to start the Barrett esophagus determination process, theCPU 22 sets the variable N, specifically, sets the variable N at 0.

In the next step S8, the CPU 22 compares the variable N with apredetermined constant MAX_N, more specifically, the maximum framenumber for which the process of determining whether or not the patienthas the Barrett esophagus is to be executed. Then, if the comparisonresult indicates N>MAX_N, the CPU 22 ends the process. The presentembodiment thus avoids determining whether or not the patient has theBarrett esophagus, for images with frame numbers following the presetmaximum frame number.

On the other hand, if step S8 results in the opposite comparison result,that is, N≦MAX_N, the process proceeds to step S9, where the imageextracting block 41 extracts an image with a frame number=COUNT+N. Thatis, an image is extracted which is located temporally N frames after theimage in which the EG junction 35 is detected (At this time, N is 0,that is, the initial value. Accordingly, the Barrett esophagusdetermination process is executed on the basis of the image in which theEG junction 35 has been detected. As is apparent from the subsequentprocess, whether or not the patient has the Barrett esophagus issequentially executed on images picked up temporally after the one inwhich the EG junction 35 has been detected).

Then, in step S10, the EG junction detection block 45 executes a processof detecting the EG junction 35 in the image with that frame number.

In the next step S11, the epithelium boundary detection block 44executes a process of detecting the epithelium boundary 34 in the imagewith that frame number as a process of detecting the second feature. Theprocess of detecting the epithelium boundary 34 corresponds to, forexample, the process from step S1 to step S4 shown in FIG. 4 of JapanesePatent Application No. 2004-360319. Specifically, since the squamousepithelium in the esophagus side has a color tone different from that ofthe columnar epithelium in the stomach side as described above, thecoordinates of the epithelium boundary 34 can be calculated (detected)by executing an edge process and a thinning process on endoscopic imagedata and then joining a generated sequence of points for the boundarytogether to obtain a coordinate point sequence along the boundary.

In the next step S12, the Barrett esophagus determination block 46 usesthe point sequence for the line indicating the EG junction 35 detectedin step S10 and the point sequence for the line indicating theepithelium boundary 34 detected in step S11 to determine whether or notthe condition determination target site in the picked-up image is theBarrett esophagus. The Barrett esophagus determination block 46constitutes a condition determination section that make a determinationfor the condition of a living body on the basis of the detection resultfrom the epithelium boundary detection section 44 to output adetermination.

Specifically, a process described in connection with a Barrett esophagusdetermination process shown in FIG. 13 described below makes it possibleto determine whether or not the patient has the Barrett esophagus.

In step S13, the Barrett esophagus determination block 46 passes thedetermination of whether or not the target site is the Barrett esophagusand the frame number to the display processing block 43. The displayprocessing block 43 extracts image data indicated by the specified framenumber from an internal buffer (not shown) and superimposes thedetermination on the image data. The image data is sent to the monitor4, which displays the image together with the determination on thedisplay screen.

For example, if the target site is determined to be the Barrettesophagus, then as shown in FIG. 6B, for example, “suspected Barrettesophagus” is displayed in the determination target image.

In step S14 subsequent to step S13, the variable N is incremented byone, and then the process returns to step S8. The process from step S8to step S14 is then repeated. Thus, when the variable N exceeds themaximum value MAX_N, the process is ended.

According to the present embodiment configured as described above andexecuting the process described above, to analyze images to determinewhether or not analysis target still image data constituting motionpicture data on picked-up endoscopic images shows the Barrett esophagus,the process of detecting an image having the feature of the EG junction35 is executed in order of picked-up images, the EG junction 35constituting the end points of the palisade vessels, which are presentaround the periphery of the Barrett esophagus determination site. Theprocess of detecting the feature of the epithelium boundary 34, requiredto make a determination for the Barrett esophagus condition, is thenexecuted on images following the one determined by the above process tohave the feature of the EG junction 35. Then, on the basis of thedetection result, the positional relationship between the epitheliumboundary 34 and the EG junction 35, and the like, the apparatusdetermines whether or not the target site is the Barrett esophagus. Thismakes it possible to efficiently determine whether or not the targetsite is the Barrett esophagus.

Further, determinations can be made for the Barrett esophagus and theBarrett mucosa (Barrett epithelium), which is a pre-symptom of theBarrett esophagus disease as described below. This enablesdeterminations suitable for early treatments.

Further, the present embodiment presets the maximum frame number for thecondition determination of whether or not the target site is the Barrettesophagus to avoid the condition determination of whether or not thetarget site is the Barrett esophagus, for images with frame numbersfollowing the maximum frame number. This makes it possible to preventtime from being spent on images that need not be subjected to thecondition determination of whether or not the target site is the Barrettesophagus.

That is, if images of the interior of the esophagus 33 are sequentiallypicked up starting with the mouth 31 and ending with the interior of thestomach 36, that is, the interior of the cardia 37, as shown in FIG. 2,then the image of the interior of the stomach need not be subjected tothe condition determination of whether or not the target site is theBarrett esophagus. In this case, setting the frame number of the stomachimage at MAX. N makes it possible to avoid the condition determinationof whether or not the target site is the Barrett esophagus.

Now, the process of detecting the EG junction 35 will be described withreference to FIGS. 9 to 13. Description will be given below of an imageanalysis process of detecting the EG junction 35 and then detecting theepithelium boundary 34 and making a determination for the Barrettmucosa. The image analysis process is intended to provide an apparatusand method for appropriately determining whether or not the target siteis the Barrett mucosa. The image analysis process makes it possible toappropriately determine whether or not the target site is the Barrettmucosa.

FIG. 9 shows the relevant process procedure, data generated, and thelike. The left of FIG. 9 shows the contents of the process, andinformation such as images generated is shown inside a frame in theright of the figure.

When the image analysis process is started, in the first step S21, anedge extraction process is executed on a process target image. The edgeextraction process generates an edge image by for example, applying abandpass filter to a G color component image in an RGB image.

The edge extraction technique based on the bandpass filter is wellknown. An edge image may also be generated using a luminance componentof the processing target image. If not only the edge of the vessel butalso the edge of another shape (contour) is extracted, the vessel edgealone can be extracted by applying the bandpass filter to the Rcomponent of the processing target image to exclude the edge of theextracted shape.

Steps S21 to S26 in FIG. 9 are used for a processing sectioncorresponding to the stomach/esophagus detection process in step S4 inFIG. 8.

In the next step S22 in FIG. 9, a binarization is executed on the edgeimage to generate a binarized image. The binarization in accordance withthe present embodiment compares the pixel value of each pixel in theedge image with a specified threshold to determine the value of eachpixel in the binarized image to be 0 or 1.

In the next step S23, a well-known thinning technique is applied to thebinarized image to execute a thinning process to generate a thinnedimage.

In the next step S24, a palisade vessel extraction process of extractingthe palisade vessels inherent in the esophagus 33 is executed on thethinned image. Extracted palisade vessel information is saved. Aflowchart of this process is shown in FIG. 11 (this will be describedbelow).

In the next step S25, the coordinates of the end points of the palisadevessels saved in the palisade vessel extraction process are acquired. Instep S26, a boundary line generation process of connecting a sequence ofend point coordinate points together with a segment is executed togenerate (acquire) boundary line information. FIG. 10A shows theboundary line information generated by the process, more specifically,the palisade vessel end point boundary.

Moreover, in step S27, a boundary line image is generated which containsthe boundary line information (palisade vessel end point boundary)acquired by the boundary line image generation process, and a darkportion and the epithelium boundary 34, which have already beenacquired. This image is shown in FIG. 10B.

In the next step S28, the Barrett esophagus determination process isexecuted, that is, whether or not the target site is the Barrettesophagus or the Barrett mucosa, on the basis of already acquiredinformation on the positional relationship with the epithelium boundary34 between the squamous epithelium and the columnar epithelium. Thisprocess will be described below in detail with reference to FIG. 13.

As described above, determinations are made for the Barrett esophagus orthe Barrett mucosa and displayed to finish the process.

Now, the palisade vessel extraction process in step S24 in FIG. 9 willbe described with reference to FIG. 11.

When the palisade vessel extraction process is started, in the firststep S31, unprocessed segments are acquired from the thinned image. Anexample of the corresponding image is shown in FIG. 12A.

In the next step S32, the number of pixels in each segment is calculatedto be a segment length L. In the next step S33, the calculated segmentlength L is compared with a predetermined threshold thre1 to determinewhether the former is greater or smaller than the latter. In thedetermination process, if L>ther1, the process proceeds to the next stepS34. If L≦ther1, that segment is determined not to be the palisadevessel. The process then shifts to step S41. In the present embodiment,for example, ther1=50.

In step S34, the number C of branching and intersecting points in eachsegment and the number B of bending points in each segment arecalculated. In step S35, the numbers are compared with a predeterminedthreshold E. When C≦Cth and B<ε, the process proceeds to the next stepS36. When C>Cth or B≧ε, that segment is determined not to be theextraction target palisade vessel but a dendritic vessel. The processthen shifts to step S41. In the present embodiment, Cth=0 and ε=3.

In step S36, one of the two end points of the segment which is closer tothe already acquired image dark portion is acquired. In step S37, avector v connecting the end point and the center of the dark portiontogether is calculated.

In the next step S38, the angle θ between the vector v and a straightline connecting the segment start point and the segment end pointtogether is calculated. In the next step S39, the apparatus determineswhether the calculated angle θ is greater or smaller than a thresholdthre2.

If the determination in step S39 indicates that θ<thre2 (for example, θ1in FIG. 12B), the process proceeds to step S40. In contrast, if θ≧thre2(for example, θ2 in FIG. 12B), that segment is determined not to be thepalisade vessel. The process then shifts to step S41. In the presentembodiment, thre2=45°.

In step S40, those of the segments extract in step S31 which meet thedetermination conditions in step S39 are determined to be the palisadevessels. The information (the segment length L, the branching andintersecting point count C of the segment, the bending point count B,the coordinate point sequence of the segment, the coordinates of the endpoint, and the angle θ) on those segments is saved as palisade vesselinformation. This enables the palisade vessels to be extracted as shownin FIG. 12C.

As described above, the process from step S31 to step S35 constitutes aprocessing section in which the EG junction detection block 45determines the segment to be the palisade vessel taking into account thebranching point count, intersecting point count, and bending point countof the segment obtained by executing the thinning process on the frameimage data. The process from step S36 to step S39 constitutes aprocessing section in which the EG junction detection block 45determines the segment to be the palisade vessel taking into account theangle between the segment connecting the opposite ends of the segmentobtained by executing the thinning process on the frame image data andthe vector connecting the dark portion center of the image dark portionof the frame image to one of the opposite ends which is closer to theimage dark portion.

In step S41, the presence or absence of any unprocessed segment isdetermined. If there remains any unprocessed segment, the process loopsback to step S31. If all the segments have been processed, the processis ended.

In steps S36 to S39, a matched filter may be used to extract onlyvessels extending toward the dark portion.

Now, the Barrett mucosa determination process in step S28 in FIG. 9 willbe described with reference to FIG. 13. In step S51, the boundary lineimage generated by the above boundary line image generation process isacquired. This corresponds to FIG. 10B.

In the next step S52, the entire image is divided by a predeterminednumber of, that is, N radial lines. FIG. 10C shows that the image isdivided with N set at, for example, 8.

In the next step S53, a variable i indicating the ith radial line [i] isset at an initial value of 1.

In the next step S54, a point P1 at which the ith radial line [i]crosses the epithelium boundary 34 and a point P2 at which the ithradial line [i] crosses the boundary formed are calculated. FIG. 10Cshows an image showing the calculated points P1 and P2.

In the next step S55, the distance Q[i] between the points P1 and P2 iscalculated.

In the next step S56, the apparatus determines whether all the radiallines have been processed. That is, the apparatus determines whether ornot i is equal to the radial line count N. If i has not reached N, thenin step S57, i is incremented by one. The process then returns to stepS54 to execute a process similar to that described above. If all theradial lines have been processed, the process proceeds to step S58.

Once the distance Q[i] between the points P1 and P2 is calculated forall the radial lines, that is, the N radial lines, the N distances Q[i]are used to calculate a variance σ in step S58.

In the next step S59, the apparatus determines whether the variance σ isgreater or smaller than a predetermined threshold thre3. If σ>thre3, theprocess proceeds to step S60. In contrast, if σ≦thre3, the image isdetermined not to show the Barrett mucosa. The process is then ended. Inthe present embodiment, thre3=5.

In step S60, if the image acquired in step S51 meets the determinationcondition in step S59, the image is determined to show the Barrettmucosa. The determination is, for example, displayed, announced, orsaved, and the process is then ended.

Whether or not the target site is the Barrett mucosa (Barrettepithelium) can be accurately determined in accordance with the processshown in FIGS. 9 to 13.

That is, the process detects each of the EG junction 35 and theepithelium boundary 34, and on the basis of the detection results,determines whether or not the target site is the Barrett mucosa. Thisenables appropriate and accurate determinations.

The process shown in FIG. 13 may be partly changed as described below sothat substantially quantitative determinations can be made for theBarrett mucosa and the Barrett esophagus by calculating (estimating) theradius (diameter) of the esophagus 33 in the image and using a knownstatistical value for the radius.

A process of calculating the distance (defined as R[i] forsimplification) between the dark portion center O and the point P1 (orbetween the dark portion center O and the point P2) is executed between,for example, steps S55 and S56 in FIG. 13.

The determination process in step S56 is executed, and not only thedistance Q[i] between the points P1 and P2 but also the distance R[i] iscalculated for all the radial lines [i]. Subsequently, instead ofcalculating the variance σ of the distances Q[i] in step S58 in FIG. 13,the average value Rav of the distances R[i] is calculated. The averagevalue Rave is determined to be an evaluation value (estimation value)for the radius near the epithelium boundary 34 in the esophagus 33.

A statistical radius value Rs (cm) for the esophagus 33 of a normaladult or a person having a body type similar to that of the patient ispre-stored in the memory or the like. The average value Rav and theradius value Rs are used to evaluate the average value of the distanceQ[i] between the points P1 and P2.

The apparatus then determines whether or not the average value of thedistances Q[i] is at least 3.0 cm, and if the distance Q[i] is at least3.0 cm, determines that the target site is the Barrett esophagus.

That is, the presence or absence of the Barrett esophagus is determinedtaking into account each distance between the dark portion center O as apredetermined point and the epithelium boundary crossing each of theplurality of radial lines radially extending from the dark portioncenter O or each distance between the EG junction and the dark portioncenter O.

Further, if the average value of the distances Q[i] is, for example,about 1.5 cm, this may be determined to indicate the substantialprogress of the Barrett mucosa. Further, if the average value of thedistances Q[i] is, for example, about 0.5 cm, this may be determined tobe an initial symptom of the Barrett mucosa.

Thus, the present variation makes it possible to substantiallyquantitatively determine whether or not the target site is the Barrettesophagus, and in the case of the Barrett mucosa, to quantitativelydetermine the progress of symptoms. Then, early treatments can beachieved by for example, displaying the determination.

Alternatively, instead of the Barrett mucosa determination process shownin FIG. 9, a process such as a variation shown in the flowchart in FIG.14 may be executed.

The present variation differs from the process in the above flowchart inthat as shown in FIG. 14, an epithelium boundary palisade vessel imagegeneration process in step S61 is executed in place of the vessel endpoint extraction process in step S25 in FIG. 9, the boundary linegeneration process in step S26 in FIG. 9, and the boundary line imagegeneration process in step S27 in FIG. 9. An epithelium boundarypalisade vessel image generated by this process is used to execute aBarrett mucosa determination process in step S62.

In the epithelium boundary palisade vessel image generation process instep S61, an epithelium boundary palisade vessel image is generated asshown in FIG. 15A, the image containing the palisade vessels acquired bythe palisade vessel extraction process, and the dark portion andepithelium boundary already acquired.

In the next step 62, the epithelium boundary palisade vessel imagegenerated in the last step S61 is used to execute the Barrett mucosadetermination process.

FIG. 16 shows a flowchart of the Barrett mucosa determination process.

As shown in FIG. 16, in the first step S63, an image containing theepithelium boundary 34 already acquired and the palisade vessels isacquired.

In the next step S64, the number J of palisade vessels crossing theepithelium boundary line is initialized, that is, J is set at 0.

In the next step S65, a processing target vessel is acquired from the Qpalisade vessels. Moreover, in the next step S66, the apparatusdetermines whether or not the processing target vessel crosses theepithelium boundary 34. If the processing target vessel crosses theepithelium boundary 34, the process proceeds to the next step S67 to add1 to the palisade vessel count J. If the processing target vessel doesnot cross the epithelium boundary 34, the process returns to step S65 toacquire the next processing target vessel. Then, the same process asdescribed above is repeated.

In step S68 subsequent to step S67, the apparatus determines whether ornot all the palisade vessels have been processed. If there remains anyunprocessed palisade vessel, the process returns to step S65 to repeatthe same process as described above. In contrast, if all the palisadevessels have been processed, the process proceeds to the next step S69.

FIG. 15B shows an example of an image for which the number J of palisadevessels crossing the epithelium boundary 34 is calculated. In FIG. 15B,the number Q of palisade vessels is 7, and 6 (=J) of these palisadevessels cross the epithelium boundary 34.

In step S69, the apparatus determines whether J/Q is larger or smallerthan a predetermined threshold thre4. If J/Q>thre4, the process proceedsto step S70. In contrast, if J/Q≦thre4, the image is determined not toshow the Barrett mucosa. The process is then ended. In the presentembodiment, thre4=0.5.

In step S70, if the image acquired in step S63 meets the determinationconditions in steps S66 and S69, the image is determined to show theBarrett mucosa. The determination is, for example, displayed on themonitor 4, and the process is then ended.

The present variation makes it possible to determine whether or not thetarget site is the Barrett mucosa depending on how many palisade vesselend points are present inside the epithelium boundary 34.

As described above, according to the present embodiment, to analyze alarge amount of still image data constituting motion picture data onendoscopic images to determine whether or not the target site is theBarrett esophagus, a process is executed which involves detecting animage having the feature of the EG junction 35, a first feature sitepresent around the periphery of a Barrett esophagus determination targetsite and constituting the stomach-side end points of the palisadevessels. Then, for example, a process of detecting the epitheliumboundary 34, a second feature site, is executed on an image followingthe one detected by the above process, to determine whether or not thetarget site is the Barrett esophagus. This makes it possible to make anefficient condition determination, that is, to efficiently determinewhether or not the target site is the Barrett esophagus. This iseffective for reducing the effort required for manual extractionoperations.

Further, the EG junction 35, set to be the first feature site thefeature of which is detected first as described above, is also utilizedto make a determination for the Barrett esophagus condition. Thisenables the feature detection to be effectively utilized.

A determination can also be made for the Barrett mucosa (Barrettepithelium), a pre-symptom of the Barrett esophagus disease. Thisenables determinations suitable for early treatments and the like.

It is also possible to avoid the determination of whether or not thetarget site is the Barrett esophagus, for images that do not require thedetermination.

The present embodiment has been described in conjunction with motionpicture data. However, the present invention may be applied to aplurality of consecutive still image data for one examination (this alsoapplies to the other embodiments and the like).

Second Embodiment

Now, a second embodiment will be described with reference to FIGS. 17 to26. In the above first embodiment, to determine whether or not thetarget site is the Barrett esophagus, the process of detecting the EGjunction 35 is first executed to detect an image containing the EGjunction 35.

The process of detecting the EG junction 35 imposes a heavy load,reducing processing speed. Accordingly, the processing speed or thedetection speed needs to be improved. A cardia detection process ispossible which can be executed more quickly than the detection of the EGjunction 35 and which deals with the biological site expected to beaccurately detected.

The present embodiment focuses on this to improve the processing speedand the like.

The configuration of the hardware of an image processing apparatus inaccordance with the present embodiment is similar to that in accordancewith the first embodiment; the configuration can be described withreference to FIG. 1. FIG. 17 shows the functional configuration of theCPU 22 based on a processing program in accordance with the presentembodiment. In the configuration shown in FIG. 17, a cardia detectionblock 47 is further provided in the image analysis block 42, included inthe configuration shown in FIG. 4. The cardia detection block 47constitutes a first biological feature detection section that detectsthe first biological feature.

FIG. 18 shows a flowchart of a process in accordance with the presentembodiment. In accordance with the flowchart, a Barrett esophagusdetermination is made, and for example, the determination is displayed.

The cardia detection block 47, for example, detects a dark portion andthus detects the cardia 37 in the image data on the basis of the shapeof the detected dark portion and the degree of rapidity of a change inlightness near the edge of the dark portion. This will be describedbelow in detail with reference to FIG. 19.

The process procedure shown in FIG. 18 is different from that shown inthe flowchart in FIG. 8 in that a process of detecting the cardia 37 instep S4′ is executed in place of the process of detecting the EGjunction 35 in step S4 and that a process of determining the presence ofthe cardia 37 in step S5′ is executed in place of the process ofdetermining the presence of the EG junction 35 in step S5, followingstep S4.

Further, instead of extracting the image with the frame number=COUNT+Nin step S9 of the process procedure in FIG. 8, an image with a frameimage=COUNT−N is extracted as shown in step S9′ in FIG. 18.

That is, in the first embodiment, the EG junction 35 is located aroundthe periphery of the Barrett esophagus determination target site.Accordingly, an image obtained temporally after the site is detected inthe original image is examined to obtain an image showing an area closerto the periphery of the site. This meets image pickup conditions for animage pickup operation performed while inserting the distal end 14 ofthe endoscope 6.

In contrast, as seen in FIG. 2, the cardia 37 is a site serving as aninlet to the stomach through which the endoscope enters the stomachafter having passed through the EG junction 35 and the epitheliumboundary 34. Consequently, a frame image is extracted which is locatedtemporally N frames before the image in which the cardia was detected.Images picked up before the above image are sequentially subjected tothe determination of whether or not the image shows the Barrettesophagus.

According to the present embodiment, after the detection of the cardia37, which imposes a lighter processing load than the detection of the EGjunction 35, the Barrett esophagus determination process is executed onthe basis of the image in which the cardia 37 has been detected. Thisenables the Barrett esophagus determination to be achieved in a shortertime.

This makes it possible to shift more efficiently to the Barrettesophagus determination process, allowing effects similar to those ofthe first embodiment to be exerted in a shorter time.

Now, the process of detecting the cardia 37 will be described withreference to FIGS. 19 to 22. FIG. 19 shows a process flow in which theclosed cardia is detected, as well as data used or generated during theprocess.

When the process of detecting the cardia 37 is started, an edgedetection process is executed on a processing target image as shown instep S71 to generate an edge image.

In the present embodiment, the edge extraction process generates an edgeimage by applying a bandpass filter to an R color component image.

The edge extraction technique based on the bandpass filter is wellknown. An edge image may also be generated using a luminance componentof the processing target image.

In the next step S72, a binarization is executed on the edge image togenerate a binarized image. The binarization in accordance with thepresent embodiment compares the pixel value of each pixel in the edgeimage with a specified threshold to determine the value of each pixel inthe binarized image to be 0 or 1.

In the next step S73, a well-known thinning technique is applied to thebinarized image to execute a thinning process to generate a thinnedimage. FIG. 20A shows an example of a generated thinned image.

In the next step S74, a branching and intersecting point calculationprocess of calculating branching and intersecting points for all thethin lines in the thinned image. FIG. 20B shows an example of branchingand intersection points calculated for the thinned image in FIG. 20A.FIG. 20B shows that the number Nc of branching and intersecting pointsis 5.

The coordinates of the branching and intersecting points calculated bythe branching and intersecting point calculating process in step S74 aresaved as branching and intersecting point information.

In the next step S75, a concentration level calculation process isexecuted for calculating the concentration level of the branching andintersecting points on the basis of the coordinate values of thebranching and intersecting points. Thus, concentration level informationis calculated.

The concentration level information will be described below withreference to the flowchart in FIG. 21. In the first step S77, thecoordinate values of the Nc branching and intersecting points areacquired. In the next step S78, calculation is made of the variance σxof the x coordinates of the Nc branching and intersecting points and thevariance σy of the y coordinates of the Nc branching and intersectingpoints. In the next step S79, the variances σx and σy are saved asconcentration level information. The process is then ended.

In the concentration level calculation process, instead of thevariances, standard deviations, coefficients of variations, averagevalues of the distances between each of the Nc branching andintersecting points and the centroid, or the like may be determined toobtain concentration level information.

Referring back to FIG. 19, the concentration level informationcalculated by the concentration level calculation process in step S75 isused to execute a closed cardia determination process in the next stepS76.

As shown in step S76 a in FIG. 22, the closed cardia determinationprocess make a determination by comparing the branching and intersectingpoint count Nc with a predetermined threshold thre_N and compares theconcentration level information (σx, σy) with thresholds thre_x andthre_y, respectively.

If the conditions Nc>thre_N, σx<thre_x, and σy<thre_y in step S76 a aredetermined to be met, the target site is determined to be the closedcardia 37 as shown in step S76 b. On the other hand, the conditions instep S76 a are determined not to be met, that is, Nc≦thre_N, σx≧thre_x,or σy≧thre_y, the target site is determined to not to be the closedcardia 37 as shown in step S76 c.

The cardia determination process is thus executed, and the cardiadetection process shown in FIG. 19 is then ended.

This enables the cardia 37 to be detected.

As described above, the present embodiment first executes the process ofdetecting the cardia 37, and if the cardia 37 is detected, makes adetermination for the Barrett esophagus, the determination target, forpreviously picked-up images. This enables the Barrett esophagusdetermination to be efficiently made even with a large amount of imagedata.

Further, if the Barrett esophagus determination process is executed asdescribed in the first embodiment, the Barrett mucosa determination canalso be made. This determination is effective for early treatments.

FIG. 23 shows a flowchart process of cardia detection in accordance witha variation.

The present variation executes an edge component generation anglecalculation process in step S81 which is intended to detect the opencardia, in place of the branching and intersecting point calculationprocess in step S74 and the next concentration level calculation processin step S75, in the process flowchart shown in FIG. 19. The presentvariation then executes an open cardia determination process ofdetecting (determining) the open cardia in step S82 on the basis ofgeneration angle information calculated by the edge component generationangle calculation process.

Steps S71 to S73 in the process shown in FIG. 23 are the same as thoseshown in FIG. 19.

An edge extraction process is executed on a processing target image asshown in step S71 to generate an edge image. The binarization in stepS72 is further executed on the edge image to generate a binarized image.The thinning process in step S73 is further executed to generate athinned image shown in FIG. 24A.

Then, an image containing a dark portion is provided which has beensubjected to a dark portion binarization using a dark portion extractionthreshold in order to allow the image dark portion to be extracted. Theimage is superimposed on the thinned image acquired by the thinningprocess to obtain an image shown in FIG. 24B. The edge componentgeneration angle calculation process in step S81 is then executed on theresulting image to calculate generation angle information on a high edgeangle.

An open cardia determination process in step S82 is then executed todetermine whether or not the target site is the open cardia.

FIG. 25 is a flowchart showing the details of the edge componentgeneration angle calculation process in step S81 in FIG. 23.

In the first step S83, one feature point in the dark portion in theimage is selected. In the present embodiment, the feature point in thedark portion in the image selected for calculation is, for example, thecentroid of the dark-portion.

In the next step S84, the image is divided into a plurality of, forexample, M pieces around the calculated feature point such as thecentroid or center point in the circumferential direction, using radiallines.

In the next step S85, one of the thin lines in the thinned image shownin FIG. 24A described above, that is, a segment i, is extracted(acquired).

In the next step S86, the angle θ[i] through which the extracted segmenti is present is calculated. That is, the number Ni of areas in which thesegment i is present is counted, and on the basis of the count, theangle θ[i] through which the extracted segment i is present iscalculated by θ[i]=Ni×(360/M)°.

An example of the calculation is shown in FIG. 24C. FIG. 24C shows thatthe number Ni of areas in which the segment i is present is 6, the rangeof areas being enclosed by a parting line of 0° and a parting line of270°. That is, in FIG. 24C, the area (shaded part) in which the edge ispresent spans 270°.

In the next step S87, the apparatus determines whether or not thereremains any unprocessed segment. If there remains any unprocessedsegment, the process returns to step S85 to acquire the unprocessedsegment. Then, a process similar to that described above is executed.

On the other hand, if all the segments have been processed, the edgecomponent generation angle calculation process is ended. Referring backto FIG. 23, angle information on the angle θ[i] generated by the edgecomponent generation angle calculation process in step S81 is used toexecute the open cardia determination process in step S82 to determinewhether or not the target site is the open cardia.

The open cardia determination process compares the angle θ[i] with apredetermined threshold thre5, for example, as shown in step S82 a inFIG. 26. That is, the apparatus determines whether a conditionθ[i]>thre5 is met. If the condition θ[i]>thre5 is met, the apparatusdetermines that the edge corresponds to the open cardia as shown in stepS82 b. In contrast, if the condition is not met, the apparatusdetermines that the edge does not correspond to the open cardia. Thus,the process of detecting the cardia is ended.

This enables the open cardia to be detected.

Then, the process of detecting the Barrett esophagus is executed on theimage in which the cardia has been detected. This enables an efficientBarrett esophagus determination.

Third Embodiment

Now, a third embodiment of the present invention will be described withreference to FIGS. 27 to 29. The configuration of the hardware of animage processing apparatus in accordance with the present embodiment issimilar to that in accordance with the first embodiment; theconfiguration can be described with reference to FIG. 1. FIG. 27 showsthe functional configuration of essential sections provided by the CPU22 executing a processing program in accordance with the presentembodiment. In the configuration shown in FIG. 27, a processingcontinuation determination block 48 is further provided in the imageanalysis block 42, included in the configuration shown in FIG. 4. Theprocessing continuation determination block 48 constitutes a biologicalfeature detection section that detects the first biological feature.

The processing continuation determination block 48 determines whether ornot a point sequence for the epithelium boundary line detected by theepithelium boundary detection block 44 is present. The processingcontinuation determination block 48 further controls the operation ofthe image analysis block 42 in accordance with the above determination.

FIG. 28 shows a flowchart of a process procedure in accordance with thepresent embodiment. In accordance with the flowchart, a Barrettesophagus determination is made, and for example, the determination isdisplayed.

The process method in accordance with the flowchart shown in FIG. 28corresponds to the process procedure in accordance with the firstembodiment shown in FIG. 8 and in which instead of the process ofdetecting the EG junction 35, an easier process of detecting theepithelium boundary 34 is executed.

First, an image in which the epithelium boundary 34 is detected isretrieved. Once the image in which the epithelium boundary 34 isdetected cab be retrieved, the Barrett esophagus determination processis executed.

The process procedure will be described with reference to the flowchartin FIG. 28. The initial steps S1 to S3 are the same as those in theflowchart in FIG. 8 and will thus not be described.

In step S3, an image with the frame number COUNT is extracted, and inthe next step S91, a process of detecting the epithelium boundary 34 isexecuted.

In the next step S92, for the epithelium boundary 34 subjected to thedetection process, the processing continuation determination block 48determines whether or not a point sequence for a line indicating theepithelium boundary 34 is obtained in step S91 to determine whether ornot the epithelium boundary 34 is present.

If the apparatus determines in step S92 that the epithelium boundary 34is not present, the process proceeds to step S6 to increment the framenumber variable COUNT by one. The process then returns to step S2 tocontinue the process from step S2 to step S92, that is, the analysisoperation of detecting the epithelium boundary 34.

On the other hand, if the apparatus determines in step S92 that theepithelium boundary 34 is present, the process shifts from the analysisoperation of detecting the epithelium boundary 34 to step S93, that is,the analysis operation of making a determination for the Barrettesophagus.

In step S93, the EG junction detection block 45 executes the process ofdetecting the EG junction 35 starting with the image with that framenumber.

After the process of detecting the EG junction 35, in the next step S94,the Barrett esophagus determination block 46 uses the point sequence forthe line indicating the EG junction 35 detected in step S93, and thepoint sequence for the line indicating the epithelium boundary 34detected in step S91, to determine whether or not the target site in thepicked-up image is the Barrett esophagus.

Upon determining that the target site is the Barrett esophagus, theBarrett esophagus determination block 46 passes the determination andthe frame number to the display processing block 43 in step S95. Thedisplay processing block 43 extracts the image with the specified framenumber from the buffer and superimposes the determination on the imagedata. For example, the display processing block 43 provides such adisplay as shown in FIG. 7.

In the next step S96, the COUNT is incremented by one. In the next stepS97, an image with the next frame number (=COUNT) is newly acquired. Theprocess of detecting the epithelium boundary 34 is then executed on theimage.

In the next step S98, the apparatus determines whether or not theepithelium boundary 34 is present on the basis of the precedingdetection process.

In step S98, the processing continuation determination block 48determines whether or nor the point sequence for the line indicating theepithelium boundary 34 is obtained in the preceding step S97, todetermine whether or not the epithelium boundary 34 is present.

If the apparatus determines in step S98 that the epithelium boundary 34is not present, the process loop from step S93 to step S98 is stopped,that is, the analysis operation for the Barrett esophagus determinationprocess is stopped to end the process.

On the other hand, if the epithelium boundary 34 is determined to bepresent, the process returns to step S93 to execute the process ofdetecting the EG junction 35 to continue the Barrett esophagusdetermination process. In this manner, if the presence of the epitheliumboundary 34 is detected, the process in the process loop is repeated.The process is ended when the presence of the epithelium boundary 34fails to be detected again. That is, the biological conditiondetermination is performed on frame images in which the epitheliumboundary has been detected.

The present embodiment, operating as described above, first executes theprocess of detecting the epithelium boundary 34, and when the epitheliumboundary 34 is detected through the detection process, shifts to theprocess of determining the presence or absence of the Barrett esophagus.Then, when the presence of the epithelium boundary 34 fails to bedetected again, the process is ended. This enables an efficient Barrettesophagus determination.

That is, the present embodiment provides the processing continuationdetermination section to perform the following control. Only the imageof the periphery of the epithelium boundary 34 is detected, which isrequired to make a determination for the Barrett esophagus condition.Then, when the epithelium boundary 34 fails to be detected in the imageagain, the Barrett esophagus determination process is ended. Thisenables the image required for the Barrett esophagus determinationprocess to be extracted to allow the Barrett esophagus determinationprocess to be executed, without the need for much time and effort.

That is, the present embodiment allows the Barrett esophagusdetermination process to be executed in a shorter time and with lesseffort than the first embodiment.

In the present embodiment, after the epithelium boundary 34 is detectedin a frame image, the next epithelium boundary detection target image isa frame image temporally and consecutively following the above frameimage. However, the next detection target image to be acquired may be atemporally preceding frame image depending on, for example, the temporaldirection in which images are picked up.

Alternatively, the next frame image to be acquired may be specified bydefining the intervals at which consecutive frame images are acquired asN (N is a natural number of 1, 2, 3, . . . ) and incrementing COUNT toCOUNT+N in step S97.

Now, a variation of the present embodiment will be described. When theapparatus determines whether or not each image shows the Barrettesophagus, the picked-up image may be erroneously determined not to showthe Barrett esophagus under the effect of noise, halation, a temporalvariation in light quantity, shading, or the like contained in the imagedata, though the image actually shows the Barrett esophagus.

Thus, the present variation solves this problem by the process procedureshown in FIG. 29. Steps S1 to S94 in the process procedure shown in theflowchart in FIG. 29 are the same as those in the process procedureshown in FIG. 28 (however, in FIG. 29, a frame number variable Nb isadditionally used and thus initialized to zero in step S1).

In FIG. 28, the determination in step S94 is displayed in step S95.However, in the present variation, after the determination process instep S94, the frame number is changed to the next one. Then, the processof detecting the epithelium boundary 34, the process of detecting the EGjunction 35, and the like are executed to allow the Barrett esophagusdetermination process to be executed.

Then, when the current image is found not to contain the epitheliumboundary 34, the process totally determines whether or not the targetsite is the Barrett esophagus on the basis of all the determinations ofwhether or not the target site is the Barrett esophagus. The totaldetermination is then displayed.

The process procedure will be described below with reference to FIG. 29.Steps S1 to S94 in the process procedure are the same as those shown inFIG. 28 (however, as described above, in step S1, another frame numbervariable Nb is initialized to zero) and will not be described below.

In step S94, the apparatus determines whether or not the target site isthe Barrett esophagus on the basis of the position at which theepithelium boundary 34 is present and which has been determined in stepS92 as well as the result of the process of detecting the EG junction 35in step S93. The Barrett esophagus determination is temporalily stored,and in the next step S101, the variable of the frame number Nb isincremented by one. In the next step S102, an image with the framenumber Nb incremented by one, that is, the frame number COUNT+Nb, isextracted.

In the next step S103, the process of detecting the epithelium boundary34 is executed. After the detection process, the apparatus determines inthe next step S104 whether or not the epithelium boundary 34 is present.If the epithelium boundary 34 is present, the process returns to stepS93 to execute the process of detecting the EG junction 35 to execute,for example, the process of determining whether or not the target siteis the Barrett esophagus as described above.

On the other hand, upon determining that the epithelium boundary 34 isnot present, the process shifts to step S105 to acquire all thedeterminations for the Barrett esophagus made during the process fromstep S93 to step S104.

In the next step S106, the process totally determines whether or not thetarget site is the Barrett esophagus on the basis of all the Barrettesophagus determinations. In the next step S107, the total determinationis then displayed so as to be superimposed on the image. The process isthen ended.

The Barrett esophagus determination block 46 totally determines in stepS106 whether or not the target site is the Barrett esophagus as follows.

For example, the Barrett esophagus determination block 46 calculates theratio Na/Nb of the number Na of images determined to show the Barrettesophagus to the number Nb of images subjected to the determination ofwhether or not the target site is the Barrett esophagus. If the ratioNa/Nb is greater than 0.8, the Barrett esophagus determination block 46determines that the image pickup target is the Barrett esophagus. Instep S107, the Barrett esophagus determination block 46 superimposesinformation “Suspected Barrett esophagus” on all the Nb images used forthe Barrett esophagus determination.

The present variation, performing the process operation as describedabove, exerts effects similar to those of the third embodiment. Thepresent variation further makes a total determination using informationon the determination of whether or not each of a plurality of imagesshows the Barrett esophagus. This makes it possible to very reliablydetermine whether or not the target site is the Barrett esophagus.

In the above description, image processing is executed on endoscopicimages picked up by inserting the endoscope 6 with the elongateinsertion portion into the living body. However, the contents of theprocess in each of the above embodiments and variations are alsoapplicable to endoscopic images picked up by a capsule endoscope that isswallowed through the mouth to pick up in vivo images.

The capsule endoscope is normally an in vivo image pickup apparatus thatconsecutively picks up still images at regular time intervals. In thiscase, after swallowed through the mouth, the capsule endoscope movesthrough the esophagus 33, the stomach, the small intestine, and thelarge bowel, while picking up images thereof, while without movingbackward. Each of the above embodiments and variations is alsoapplicable to this case.

Further, the present invention includes embodiments each obtained by,for example, partly combining the above embodiments and the liketogether.

As described above, according to the above embodiments and variations,to determine the condition of the living body, it is possible to detectthe first biological feature and then the second biological feature.This enables an efficient determination for the condition of interestsuch as the Barrett esophagus from a large amount of image data. Thatis, if the first biological feature fails to be detected, the detectionof the second biological feature is omitted. This enables an efficientimage analysis process. Therefore, even with a large amount of imagedata, an efficient determination can be made for the condition ofinterest such as the Barrett esophagus.

Now, a luminal image processing apparatus in accordance with anembodiment will be described with reference to the drawings.

Fourth Embodiment

First, with reference to the drawings, description will be given of aluminal image processing apparatus utilizing a capsule endoscopeapparatus as well as a method for the luminal image processing inaccordance with a fourth embodiment. First, with reference to FIGS. 30A,30B and 31, description will be given of the luminal image processingapparatus utilizing the capsule endoscope apparatus in accordance withthe fourth embodiment. FIGS. 30A and 30B are block diagrams showing thegeneral configuration of a capsule endoscope apparatus 101 and aterminal apparatus 107 serving as a luminal image processing apparatusin accordance with the present embodiment.

As shown in FIG. 30A, the capsule endoscope apparatus 101 using an imageprocessing method in accordance with the present embodiment comprises acapsule endoscope 103, an antenna unit 104, and an external device 105.Although described below in detail, the capsule endoscope 103 is shapedso as to be swallowed through the mouth of a patient 102 that is asubject to advance through the esophagus and the gastrointestinal tract.The capsule endoscope 103 internally has an image pickup function ofpicking up images of the esophagus and the gastrointestinal tract togenerate picked-up image information and a transmission function oftransmitting the picked-up image information to the exterior of theliving body. The antenna unit 104 has a plurality of reception antennas111 installed on the body surface of the patient 102 to receive thepicked-up image information transmitted by the capsule endoscope 103.The external device 105 is externally shaped like a box and hasfunctions of, for example, executing various processes on the picked-upimage information received by the antenna unit 104, recording thepicked-up image information, and displaying the picked-up images on thebasis of the picked-up image information. An armor of the externaldevice 105 has a liquid crystal monitor 112 and an operation portion 113on a surface thereof, the liquid crystal monitor 112 displaying thepicked-up images, the operation portion 113 being used to giveinstructions on the operation of the various functions. Further, theexternal device 105 has an alarm display LED for the amount of powerremaining in a battery serving as a driving power supply, and a powersupply switch serving as the operation switch 113.

The external device 105 is installed on the body of the patient 102, andas shown in FIG. 30B, is installed on a cradle 106 to connect to theterminal apparatus 107. For example, a personal computer is used as theterminal apparatus 107, which is a luminal image processing apparatusand serves as a cardia detection apparatus. The terminal apparatus 107comprises a terminal main body 109 having functions of processing andstoring various data, a keyboard 108 a and a mouse 108 b which are usedto input various operations, and a display 108 c that displays theresults of processing results. The basic function of the terminalapparatus 107 is to load, via the cradle 106, the picked-up imageinformation recorded in the external device 105, and to write and recordthe picked-up image information in a rewritable memory contained in theterminal main body 109 or a portable memory such as a rewritablesemiconductor memory that can be freely installed in and removed fromthe terminal main body 109, and to execute image processing such thatthe recorded picked-up image information is displayed on the display 108c. Moreover, the terminal apparatus 107 executes a cardia detectionprocess using an image processing method in accordance with anembodiment described below. The picked-up image information stored inthe external device 105 may be loaded into the terminal apparatus 107via a USB cable or the like instead of the cradle 106. The cradle 106and the like constitute an image input section via which images pickedup by the capsule endoscope 3 are inputted.

Now, the external shape and internal structure of the capsule endoscope103 will be described with reference to FIG. 31. The capsule endoscope103 is shaped like a capsule comprising an armor member 114 having aU-shaped cross section and a hemispherical cover member 114 a installedat a distal open end of the armor member 114 in a water tight manner viaan adhesive and formed of a transparent member.

In an internal hollow portion of the capsule shape, comprising the armormember 114 and the cover member 114 a and inside a central portion ofthe arc of the hemisphere of the cover member 114 a, an objective lens115 is housed in a lens frame 116 to capture an image of an observationtarget which is incident via the cover member 114 a. A charge coupleddevice (hereinafter referred to as a CCD) 117 that is an image pickupdevice is located at an image formation position of the objective lens115. Four white LEDs 118 are arranged on the same plane around the lensframe 116, in which the objective lens 115 is housed, to emit andradiate illumination light (only two LEDs are shown in the figure). Thefollowing are arranged in the hollow portion of the armor member 114 andbehind the CCD 117: a processing circuit 119 that drivingly controls theCCD 117 to executes a process of generating a photoelectricallyconverted image pickup signal, an image pickup process of executingpredetermined signal processing on the image pickup signal to generate apicked-up image signal, and a LED driving process of controlling anoperation of illuminating and non-illuminating the LEDs 118, acommunication processing circuit 120 that converts the picked-up imagesignal generated by the image pickup process executed by the processingcircuit 119 into a radio signal to transmit the radio signal, atransmission antenna 123 that transmits the radio signal from thecommunication processing circuit 120 to the exterior, and a plurality ofbutton cells 121 that supply driving power to the processing circuit 119and the communication processing circuit 120. The CCD 117, LED 118,processing circuit 119, communication processing circuit 120, andtransmission antenna 123 are arranged on circuit boards (not shown) thatare connected together via flexible boards.

The capsule endoscope 103 picks up in vivo images at predetermined timeintervals while moving through the body of the patient 102 and transmitsthe images to the external device 105. The external device 105 recordseach of the received endoscopic images in a built-in storage device. Theendoscopic image recorded in the external device 105 is transferred viathe cradle 106 to the terminal apparatus 107, where the image is storedin a storage device (not shown). The terminal apparatus 107 executes acardia detection process on the basis of the transferred and storedendoscopic image. The cardia detection process is executed by imageprocessing software, that is, a program, which executes image processingon image data on the endoscopic image. The image processing software isexecuted by a processing device such as the CPU in the terminalapparatus 107.

Image processing described below is implemented by software and may beexecuted by any of the capsule endoscope 103, the external device 105,and the terminal apparatus 107. The description below takes an examplein which the image processing is executed by the terminal apparatus 107,which uses the personal computer. In the description of the contents ofthe image processing, the size of one frame image corresponds to 3planes of ISX×ISY (1≦ISX, ISY. For example, ISX=640, ISY=480) for red(R), green (G), and blue (B). Gray scales for the pixels in each planecorrespond to 8 bits, that is, have a value of 0 to 255.

Further, the capsule endoscope 103 picks up 15 to 30 images per second(15 fps to 30 fps) to, for example, examine the esophagus. The imagepickup function is controlled such that after passing through theesophagus, the capsule endoscope 103 performs a slower image pickupoperation with the reduced number of images picked up per second. Thisis achieved by, for example, providing a timer circuit (not shown) andperforming control such that when a timer count provided by the timercircuit indicates that a predetermined time has not passed yet, a fasterimage pickup operation is performed with the increased number of imagespicked up per second and such that after the predetermined time passes,a slower image pickup operation is performed with the reduced number ofimages picked up per second.

FIG. 32 is a flowchart showing an example of the flow of a processexecuted by the terminal apparatus 107 to detect the cardia by passingthrough the EG junction on the basis of a series of obtained endoscopicimages. A series of endoscopic images picked up by the endoscopeswallowed through the subject's mouth comprise a plurality of frames. Aprocess shown in FIG. 32 is executed on each of the frames. Image dataon each endoscopic image is subjected to a preprocess such as inversegamma correction or noise removal before the process shown in FIG. 32 isexecuted.

To process the first frame of the series of images on which the processshown in FIG. 32 is to be executed, first, the frame number i is set at1 (step S201). Reference character i denotes an integer from 1 to n.

Then, image data on an image Fi with the frame number i is read from thestorage device (not shown) in the terminal apparatus 107 (step S202).The image Fi comprises three planes for R, G, and B.

On the basis of the read image data on the image Fi, a predeterminedfeature value for the endoscopic image, that is, a color tone featurevalue, in this case, an average color tone feature value μi, iscalculated (step S203). The average color tone feature value μi is theaverage value of the color tone feature values for all the pixelscontained in each image. Step S203 constitutes a feature valuecalculation step or a feature value calculation section which calculatesthe average color tone feature value, the color tone feature value foreach image Fi based on the values for all the pixels.

Then, the apparatus determines whether or not the average color tonefeature value μi exceeds a predetermined threshold Th (step S204). If avalue R/(R+G+B) described below is used as the color tone feature value,the threshold Th is, for example, 0.5.

Adjusting the value of the threshold Th makes it possible to determinewhether the read image Fi shows the vicinity of the EG junction, thatis, the boundary of the gastrointestinal tract, the central portion ofthe EG junction, or the inlet of the stomach. This will be describedbelow with reference to FIG. 33.

If the determination in step S204 is YES, that is, the average colortone feature value μi exceeds the predetermined threshold Th, thecapsule endoscope 103 is about to enter the EG junction or is passingthrough the EG junction. Consequently, the apparatus determines that theread image Fi was picked up at that time (step S205). Steps S204 andS205 constitute a boundary detection section that detects the boundaryof the gastrointestinal tract on the basis of the calculated averagecolor tone feature value μi and a determination section that theintraluminal image shows an area extending from the esophagus to thecardia on the basis of the detection result of the EG junction,corresponding to the boundary.

If the determination in step S204 is NO, that is, the average color tonefeature value μi does not exceed the predetermined threshold Th, theapparatus determines whether or not the process shown in FIG. 32 hasbeen finished on all of the series of images to be subjected to theprocess shown in FIG. 32 (step S206). Once the process is finished onall the images, the determination in step S206 is YES and the process isended. If the determination in step S206 is NO, there remains anunprocessed image and a process of changing i to i+1 is thus executed(step S207). The process from step S202 to step S204 is repeated on thenext image.

As described above, in step S205, when the average color tone featurevalue μi exceeds the predetermined threshold Th, the apparatusdetermines that the capsule endoscope 103 is passing through the EGjunction or is about to enter the EG junction. In other words, theprocess can also determine that the capsule endoscope 103 willsubsequently reach the cardia or the stomach. That is, the process canalso determine that the cardia is being detected.

Now, description will be given of the color tone of the biologicaltissue extending from the esophagus to the stomach. FIG. 33 is aschematic graph illustrating a variation in color tone among a series ofendoscopic images obtained. In FIG. 33, the axis of abscissa indicatesthe image numbers (frame numbers), along the time series, of theendoscopic images picked up along a path from the esophagus squamousepithelium through the EG junction to the stomach. The axis of ordinateindicates the color tone feature value for the endoscopic imagecorresponding to each of the image numbers.

The color tone of images picked up by the capsule endoscope 103swallowed through the subject's mouth varies as shown in FIG. 33. Thatis, the color tone varies between the squamous epithelium RA of theesophagus and the stomach portion RC of the columnar epithelium. Thecolor tone varies gradually step by step in the EG junction RB, locatedbetween the squamous epithelium RA and the stomach portion RC. In FIG.33, for example, with a color tone feature value R/(R+G+B) calculatedfrom three pixel values for R, G, and B as described below, theesophagus squamous epithelium RA has a white color tone and thus a smallcolor tone feature value. The stomach portion RC has a red color toneand thus a great color tone feature value. For the EG junction RB,located between the squamous epithelium RA and the stomach portion RC,the color tone feature value varies gradually from the white color toneto the red color tone.

Thus, when such a color tone feature value as shown in step S204 in FIG.32, which varies gradually, exceeds the predetermined threshold Th (thecolor tone changes to red), the apparatus determines that the image waspicked up when the capsule endoscope 103 was about to enter the EGjunction or was passing through the EG junction. In other words, thecardia, the boundary of the gastrointestinal tract, is detected. Thatis, the cardia is detected on the basis of the difference in color tonebetween the esophagus mucosa and the stomach mucosa. Specifically, theaverage value of the color tone feature values or the like is utilizedto make a reliable determination for the passage or the like.

Now, description will be given of a specific example of the averagecolor tone feature value μi, described with reference to FIG. 32.

FIG. 34 is a flowchart showing an example of the flow of the process instep S203 in FIG. 32 which is executed on each frame image and in whichR/(R+G+B), calculated from the three pixel values for R, G, and B, isused as the average color tone feature value μi. The process shown inFIG. 34 calculates a chromaticity rj/(rj+gj+bj) on the basis of threepixel values rj, gj, bj for R, G, and B for each of the pixels in oneframe to determine the average color tone feature value μi. Referencecharacter j denotes a number identifying a pixel in the image data oneach frame.

First, j is set at 1, val is set at 0, and count is set at 0 (stepS211). Here, val denotes a variable required to determine the sum ofcolor tone feature values, and count denotes a variable required todetermine the number of pixels used to calculate the average color tonefeature value μi.

Then, the apparatus determines in step S212 whether or not the jth pixelbelongs to a dark portion. Specifically, the values of the jth pixel ineach of an R image, a G image, and a B image are defined as rj, gj, andbj. Then, if rj≦thd, gj≦thd, and bj≦thd, that pixel is determined tobelong to the dark portion. Here, thd denotes a threshold for each colorwhich is used to determine whether or not the pixel belongs to the darkportion. In the present embodiment, thd is set at 10. If the jth pixelis determined to belong to the dark portion, the process proceeds tostep S216. If the jth pixel is determined not to belong to the darkportion, the process proceeds to step S213.

Then, the apparatus determines in step S213 whether or not the jth pixelis extremely bright, that is, the jth pixel belongs to a halationportion. Specifically, if rj≧thh, gj≧thh, and bj≧thh, the pixel isdetermined to be a halation pixel. Here, thh denotes a threshold foreach color which is used to determine whether or not the pixel is ahalation pixel. In the present embodiment, Th is set at 200. If the jthpixel is determined to be a halation pixel, the process proceeds to stepS216. If the jth pixel is determined not to be a halation pixel, theprocess proceeds to step S214.

In steps S212 and S213, for the R image, G image, and B image, each ofthe thresholds thd and thh has the same value for rj, gj, and bj.However, for the biological mucosa, the R image generally tends to bebrightest. Accordingly, the threshold may be set higher for rj than forgj and bj. Alternatively, the threshold may vary among rj, gj, and bj.

In step S214, val=val+rj/(rj+gj+bj)) and count=count+1 are calculated.To determine the sum of the color tone feature values, the color tonefeature value rj/(rj+gj+bj) is added to a variable val and the variablecount is incremented by one.

In step S215, the apparatus determines whether or not the process fromstep S212 to step S214 has been executed on all the pixels.Specifically, if j<ISX×ISY, then in step S216, 1 is added to the numberj identifying the pixel (j=j+1) and steps S212 to S214 are executed onthe next pixel. If j=ISX×ISY, that is, the process from step S212 tostep S214 has been executed on all the pixels, the apparatus determineswhether or not count is greater than a threshold thc (step S217). Thethreshold thc is a value indicating the number of pixels insufficientfor color tone evaluation; a value equal to or greater than thethreshold thc means that a sufficient number of pixels are present whichare effective for color tone evaluation. In step S217, if thedetermination is YES, that is, a sufficient number of pixels are presentwhich are effective for color tone evaluation, the average color tonefeature value μi is calculated by dividing the color tone feature valuesum val by the number of pixels count used to calculate the averagecolor tone feature value μi (step 218). Specifically, μi=val/count. Asdescribed above, the average color tone feature value μi is calculatedfor the pixels in the intraluminal image other than the dark portionpixels and the halation pixels.

If the determination is NO in step S217, that is, a sufficient number ofpixels are not present which are effective for color tone evaluation,that frame image is considered to be an error, that is, an abnormalimage (step S219). The average color tone feature value μi is determinedto be, for example, 0 (zero). In step S204 in FIG. 32, the apparatusdetermines that the average color tone feature value μi does not exceedthe threshold Th.

Now, variations of the present embodiment will be described.

In FIGS. 32 to 34, described above, the apparatus determines whether ornot, for example, the capsule endoscope has passed through the EGjunction or the cardia, on the basis of each frame image. However, as afirst variation, the apparatus may determine that, for example, thecapsule endoscope 103 has passed through the EG junction when thedetermination in step S4 is μi>Th for a plurality of consecutive imagesor at least a predetermined rate (for example, 80%) of the plurality ofconsecutive images.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a second variation, theprocess shown in FIG. 32 may be executed on one particular image.

Moreover, as a third variation, the moving average of the average colortone feature value μi for a plurality of consecutive images may becalculated so that the apparatus can determine whether or not, forexample, the capsule endoscope 103 has passed through the EG junction,depending on whether or not the moving average value exceeds apredetermined threshold. For example, when m=2, 3, 4, . . . , and i>=m+1(this means that m consecutive images are obtained from n images andthat (m+1) is equal to or smaller than i), the moving average value iscalculated on the basis of the average color tone feature value μi forimages F(i−m) to Fi obtained from the m consecutive images. Theapparatus then determines whether or not the moving average exceeds apredetermined threshold. Even with a very reddish intraesophageal imageor the like possibly resulting from a variation in illuminationconditions caused by a variation in observation distance, angle, or thelike, the use of such a moving average makes it possible to eliminatethe adverse effect of a slight variation in average color tone featurevalue to more accurately determine that, for example, the capsuleendoscope 103 has passed through the EG junction.

Further, as a fourth variation, in the above example, R/(R+G+B), theratio of pixel values calculated from the three pixel values for R, G,and B, is used as a color tone feature value. The fourth variation mayuse another parameter. Another parameter for the color tone featurevalue may be, for example, G/(R+G+B) or IHb (=32 log₂(R/G)), hue, orcolor saturation.

Moreover, as a fifth variation, a plurality of color tone feature valuesmay be used. For example, in step S203 in FIG. 32, R/(R+G+B) andG/(R+G+B) may be used as the ratios of pixel values calculated from thethree pixel values for R, G, and B, and calculation is made of theaverage values of these color tone feature values, that is, the averagevalue μ1 i of the color tone feature value (R/(R+G+B)) for all thepixels in each image and the average value μ2 i of the color tonefeature value (G/(R+G+B)) for all the pixels in the image. In step S204,the apparatus determines whether or not for the average values μ1 i andμ2 i, μ1 i>Th1 and μ2 i>Th2.

Further, as a sixth variation, the passage of the capsule endoscope 103through the EG junction or the like may be detected on the basis of theamount of variation in average color tone feature value. That is, theapparatus may determine, instead of whether or not the average colortone feature value obtained from each of the series of consecutiveimages exceeds a predetermined threshold, whether or nor the amount ofvariation in the average color tone feature value for two images exceedsa predetermined threshold. That is, the average color tone feature valuefor each image is compared with that for the preceding or succeedingimage. If the difference between the two average color tone featurevalues exceeds a predetermined threshold, the apparatus may determinethat, for example, the capsule endoscope 103 has moved from theesophagus into the EG junction or from the EG junction into the stomach.The apparatus determines whether or not the differential value(μi−μ(i−m1)) between the average color tone feature values μ(i−m1) andμi for the images F(i−m1) and Fi has varied by a predetermined thresholdor more. m1 is 1, 2, 3, . . . .

The color tone of the mucosa may vary owing to individual differences inmucosa color, the presence of a lesion such as the Barrett esophagus, ora variation among image pickup systems. The sixth variation thus makesit possible to determine whether or not, for example, the capsuleendoscope 103 has passed through the EG junction without undergoing theadverse effect of the individual differences or the like.

Moreover, in this case, a variation in average color tone feature valuemay be detected by calculating the differential value of the averagecolor tone feature values.

FIG. 35 is a flowchart showing an example of the flow in which avariation in average color tone feature value is detected by calculatingthe differential value of the average color tone feature values.

Image data on each image is subjected to a preprocess such as inversegamma correction or noise removal before the process shown in FIG. 35 isexecuted, as described in conjunction with the process shown in FIG. 32.The processing from step S201 to step S203 is the same as the processingfrom step S201 to step S203 shown in FIG. 32. That is, to start theprocess with the first frame, first, the frame number i is set at 1(step S201). Then, image data on the image Fi with the frame number i isread from the storage device (not shown) in the terminal apparatus 7(step S202). The average color tone feature value μi is calculated onthe basis of the read image data on the image Fi (step S203).

The apparatus determines whether or not the process has been executed onall the images, that is, the process has been finished on all the images(step S221). If the process has not been finished on all the images, thedetermination in step S221 is NO. Then, a process of changing i to i+1is executed (step S207), and the process shifts to step S202.

When the process has been finished on all the images, the determinationin step 221 is YES, and for obtained plural average color tone featurevalue μi, a moving average value f(μi) is calculated over apredetermined range, that is, over a predetermined number of consecutiveimages for smoothing (step S222). A differential value Δf(μi) iscalculated on the basis of a temporal variation in the moving averagevalue f(μi) (step S223).

The image Fi is identified and detected which corresponds to thedifferential value Δf(μi) exceeding a predetermined threshold thf (stepS224). Steps S3 to S24 constitute a detection section that detects theboundary of the gastrointestinal tract.

This enables the detection of a plurality of images for which the amountof variation in color tone exceeds the threshold. Even with individualdifferences in mucosa color or the like, the apparatus can determinewhether or not, for example, the capsule endoscope 103 has passedthrough the EG junction without undergoing the adverse effects of theindividual differences or the like.

Moreover, as a seventh variation, a standard deviation or a variance maybe used in place of the average value of color tone feature values.

For example, FIG. 36 is a graph illustrating a variation in the standarddeviation or variance of the color tone feature value for a series ofendoscopic images obtained. In FIG. 36, the axis of abscissa indicatesthe image numbers (frame numbers), along the time series, of endoscopicimages of the area extending from the esophagus squamous epitheliumthrough the EG junction to the stomach. The axis of ordinate indicatesthe standard deviation σi or variance vi of the color tone feature valuefor the endoscopic image corresponding to each of the image numbers.

The color tone of the images picked up by the capsule endoscope 103swallowed through the subject's mouth varies as shown in FIG. 33.However, the calculated standard deviation or variance of the color tonefeature value R/(R+G+B) varies as shown in FIG. 36. That is, in each ofthe images of the squamous epithelium RA of the esophagus and thestomach portion RC of the columnar epithelium, the color tone isuniform, and the standard deviation vi or variance vi of the color tonefeature value R/(R+G+B) is thus small. However, for the EG junction RB,located between the squamous epithelium RA of the esophagus and thestomach portion RC of the columnar epithelium, a greater standarddeviation σi or variance vi is observed.

Consequently, on the basis of the standard deviation σi or variance viof the color tone feature value for each image, the apparatus candetermine whether or not, for example, the capsule endoscope 103 ispassing through the EG junction.

Moreover, instead of the standard deviation σi or variance vi of thecolor tone feature value, a variation coefficient for the standarddeviation σi or variance vi (=standard deviation σi/average color tonefeature value μi) may be used.

Moreover, the above examples use the image data on all the pixels ineach frame image. However, as an eighth variation, instead of processingall the pixels, only the pixels in predetermined regions of each framemay be sampled for processing as shown in FIG. 37. FIG. 37 shows anexample of regions in each frame image 131 which are to be subjected tothe image processing in accordance with the above present embodiment andvariations.

Each frame image 131 is divided into predetermined regions. In FIG. 37,each image 131 is divided into 16 rectangular regions. The aboveprocessing is executed only on predetermined ones (R2, R3, R5, R8, R9,R12, R14, and R15) of the resulting regions, that is, only on theregions of interest (ROI). In particular, since the esophagus is aluminal organ, the regions other than the one corresponding to thecenter of the visual field may be set to be the regions of interest(ROI) in order to more accurately calculate the color tone of the mucosasurface.

Accordingly, processing only the regions of interest (ROI) reduces theamount of calculation required, enabling an increase in processingspeed.

Moreover, when only the regions of interest (ROI) are processed, theprocessing speed may further be increased by, instead of processing allthe frames, processing only the pixels in the regions of interest (ROI)in every k (k=1, 2, 3, . . . ) frames. In particular, a large number ofimages are picked up for the interior of the esophagus. Consequently,accurate determinations may be made in spite of minor decimations.

As described above, the present embodiment (including the variations)makes it possible to determine, on the basis of the color tone featurevalue for each luminal image, whether or not the image shows that thecapsule endoscope is about to enter the EG junction or is passingthrough the EG junction.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect whether or not eachimage shows that the capsule endoscope is about to enter the EG junctionor is passing through the EG junction. However, for example, anidentification function such as a well-known linear discriminationfunction may be used for the detection. Alternatively, a feature valuein accordance with another embodiment may be combined with the presentembodiment.

Fifth Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with a fifth embodiment.Endoscopic images to be processed in accordance with the presentembodiment are a series of endoscopic images picked up by the capsuleendoscope apparatus 101 as in the case of the fourth embodiment.Accordingly, the configuration of the cardia detection apparatus issimilar to that in the fourth embodiment and will not be describedbelow.

The above fourth embodiment uses the color tone feature value. Thecardia detection apparatus, a luminal image processing apparatus inaccordance with the present embodiment, is different from the fourthembodiment in that the apparatus uses brightness information on eachimage to determine whether or not, for example, the image shows that thecapsule endoscope is passing through the EG junction.

FIG. 38 is a schematic graph illustrating a variation in brightnessamong a series of endoscopic images obtained, specifically a variationin luminance among the images. In FIG. 38, the axis of abscissaindicates the image numbers (frame numbers), along the time series, ofendoscopic images of the area extending from the esophagus squamousepithelium through the EG junction to the stomach. The axis of ordinateindicates the luminance of the endoscopic image corresponding to each ofthe image numbers.

The luminance of images picked up by the capsule endoscope 103 swallowedthrough the subject's mouth indicates the brightness of the images. Theluminance varies as shown in FIG. 38. That is, the luminance variesbetween the squamous epithelium RA of the esophagus and the stomachportion RC of the columnar epithelium. Further, the luminance for the EGjunction RB, located between the squamous epithelium RA and the stomachportion RC, is different from those for the squamous epithelium RA andthe stomach portion RC. As shown in FIG. 38, for example, with theluminance calculated on the basis of the three pixel values for R, G,and B, the esophagus squamous epithelium RA exhibits a large averageluminance value except for a dark portion and a halation portion of theimage because the esophagus squamous epithelium RA is a relativelynarrow luminal organ and lies close to the mucosa wall. The stomachportion RC exhibits a relatively low luminance. The EG junction RB,located between the esophagus squamous epithelium RA and the stomachportion RC, exhibits a larger luminance value than the esophagussquamous epithelium RA because the closed cardia is viewed from thefront in the esophagus, composed of a lumen.

Thus, when the brightness information on the image, varying gradually,exceeds a predetermined threshold Th1, the apparatus determines that,for example, the endoscope is passing through the EG junction or isabout to enter the EG junction. That is, on the basis of a variation inthe brightness information on the picked-up image, the closed cardia isdetected when the capsule endoscope 103 is about to enter the EGjunction or is passing through the EG junction. Specifically, an averageluminance value or the like is used as the brightness information inorder to allow the passage or the like to be reliably determined.

Like FIG. 32, FIG. 39 is a flowchart showing an example of the flow of aprocess of detecting the cardia when the EG junction is passed; theprocess is executed by the terminal apparatus 107 on the basis of aseries of endoscopic images obtained. The process shown in FIG. 39 issubstantially similar to that shown in FIG. 32. A series of endoscopicimages picked up by the endoscope swallowed through the subject's mouthcomprise a plurality of frames. The process shown in FIG. 39 is executedon each frame. The image data on each endoscopic image is subjected to apreprocess such as inverse gamma correction or noise removal before theprocess shown in FIG. 39 is executed. In FIG. 39, steps similar to thoseshown in FIG. 32 are denoted by the same step numbers, and thedescription of these steps is simplified. Further, in the descriptionbelow, the luminance value is used as brightness information. The casein which the G or B pixel data value is used brightness information willnot be described because this case differs from the one in which theluminance value is used as brightness information only in threshold dataused and in that the determination depends on whether or not the valueof the pixel data is smaller than the threshold.

First, to start processing with the first one of a series of images tobe subjected to the process shown in FIG. 39, the frame number i is setat 1 (step S201). Then, the image data on the image Fi with the framenumber i is read from the storage device (not shown) in the terminalapparatus 107 (step S202).

An average luminance value Ii is calculated from the read image data onthe image Fi (step S233). The average luminance value Ii is the averageof the luminance values of all the pixels contained in each image. Inthis case, the luminance value I is a feature value indicating thebrightness of the image as described above. For example, the luminance Iis calculated by 0.6R+0.3G+0.1B. Step S233 constitutes a feature valuecalculation step or a feature value calculation section which calculatesthe average luminance value based on the luminance values of all thepixels in each image Fi.

Then, the apparatus determines whether or not the average luminancevalue Ii exceeds a predetermined threshold Th11 (step S234).

Adjusting the threshold Th11 makes it possible to determine whether ornot the read image Fi was picked up in the vicinity of the inlet to theEG junction or in the central portion of the EG junction.

If the determination in step S234 is YES, that is, when the averageluminance value Ii exceeds the predetermined threshold Th11, theapparatus determines that the image Fi was picked up when the capsuleendoscope 103 was about to enter the EG junction or was passing throughthe EG junction (step S205). Steps 234 and 205 constitute a boundarydetection section that detects the boundary of the gastrointestinaltract on the basis of the calculated average luminance value Ii and adetermination section that determines that the intraluminal image showsthe area between the esophagus and the cardia on the basis of thedetection result for the EG junction, corresponding to the boundary.

If the determination in step S234 is NO, that is, when the averageluminance value Ii does not exceed the predetermined threshold Th11, theapparatus determines whether or not the process shown in FIG. 39 hasbeen finished on the series of images to be subjected to the processshown in FIG. 39 (step S206). When the process has been finished on allthe images, the determination in step S206 is NO. The process is thusended. If the determination in step S206 is NO, there remains anunprocessed image. Accordingly, a process of changing i to i+1 isexecuted (step S207), and the process from step S202 to step S234 issubsequently repeated. Steps S233 to S205 constitute a detection sectionthat detects the boundary of the gastrointestinal tract.

Also in the present embodiment, in step S205, when the average luminancevalue Ii, brightness information, exceeds the threshold Th11, the imageFi was picked up when the capsule endoscope 103 was about to enter theEG junction or was passing through the EG junction. In other words, theapparatus may also determine that the capsule endoscope 103 willsubsequently pass through the closed cardia or reach the interior of thestomach. This allows the cardia to be detected.

Now, description will be given of a specific example of the averageluminance value Ii, described with reference to FIG. 39.

FIG. 40 is a flowchart showing an example of the flow of the process instep S233 which uses, as the average luminance value Ii,(0.6R+0.3G+0.1B) calculated from the three pixel values for R, G, and Bexecuted on each frame image. The process shown in FIG. 40 calculatesthe average luminance value Ii by calculating the luminance value(0.6R+0.3G+0.1B) from three pixel values rj, gj, bj for R, G, and B foreach of the pixels in each frame.

FIG. 40 includes process steps similar to those shown in FIG. 34. Asimilar process is denoted by the same step numbers and will not bedescribed below. Reference character j denotes a number identifying apixel in the image data on each frame.

First, j is set at 1, val1 is set at 0, and count1 is set at 0 (stepS241). Here, val1 denotes a variable required to determine the sum ofbrightness feature values, and count1 denotes a variable required todetermine the number of pixels used to calculate the average luminancevalue Ii.

Then, the apparatus determines in step S212 whether or not the jth pixelbelongs to the dark portion. If the jth pixel is determined to belong tothe dark portion, the process proceeds to step S216. If the jth pixel isdetermined not to belong to the dark portion, the process proceeds tostep S213.

Then, the apparatus determines in step S213 whether or not the jth pixelis extremely bright, that is, the jth pixel belongs to a halationportion. If the jth pixel is determined to be a halation pixel, theprocess proceeds to step S216. If the jth pixel is determined not to bea halation pixel, the process proceeds to step S244.

In steps S212 and S213, for the R image, G image, and B image, each ofthe thresholds thd and thh has the same value for rj, gj, and bj.However, for the biological mucosa, the R image generally tends to bebrightest. Accordingly, the threshold may be set higher for rj than forgj and bj. Alternatively, the threshold may vary among rj, gj, and bj.

In step S244, val1=val1+(0.6rj+0.3gj+0.1bj) and count1=count1+1 arecalculated. To determine the sum of the brightness feature values, theluminance feature value (0.6rj+0.3gj+0.1bj), which is a brightnessfeature value, is added to the variable val1 and the variable count1 isincremented by one.

In step S215, the apparatus determines whether or not the process fromstep S212 to step S244 has been executed on all the pixels. If theprocess from step S212 to step S244 has not been executed on all thepixels, then in step S216, 1 is added to the number j identifying thepixel (=j+1) and steps S212 to S244 are executed on the next pixel. Ifthe process from step S212 to step S244 has been executed on all thepixels, the apparatus determines whether or not count1 is greater thanthe threshold thc (step S217). If a sufficient number of pixels arepresent which are effective for brightness evaluation, the averageluminance value Ii is calculated by dividing the variable val1 by thevariable count1 (step 218). Specifically, Ii=val1/count1.

If a sufficient number of pixels are not present which are effective forbrightness evaluation, that frame image is considered to be an error,that is, an abnormal image (step S219). The average brightness value Iiis determined to be, for example, 0 (zero). In step S234 in FIG. 39, theapparatus determines that the average brightness value Ii does notexceed the threshold Th1.

In the above description, the cardia is closed. However, the presentembodiment is also applicable to the open cardia. The cardia can bedetected on the basis of whether or not the luminance exceeds thepredetermined threshold.

Now, variations of the present embodiment will be described.

In FIGS. 39 and 40, described above, the apparatus determines whether ornot, for example, the capsule endoscope has passed through the EGjunction or the cardia, on the basis of each frame image. However, as afirst variation, the apparatus may determine that, for example, thecapsule endoscope 103 has passed through the EG junction when thedetermination in step S234 is Ii>Th11 for a plurality of consecutiveimages or at least a predetermined rate (for example, 80%) of theplurality of consecutive images.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a second variation, theprocess shown in FIG. 39 may be executed on one particular image.

Moreover, as a third variation, the moving average of the averagebrightness value Ii for a plurality of consecutive images may becalculated so that the apparatus can determine whether or not, forexample, the capsule endoscope 103 has passed through the EG junction,depending on whether or not the moving average value exceeds apredetermined threshold. For example, when m=2, 3, 4, . . . , and i>=m+1(this means that m consecutive images are obtained from n images andthat (m+1) is equal to or smaller than i), the moving average value iscalculated on the basis of the average brightness value for imagesF(i-m) to Fi obtained from the m consecutive images. The apparatus thendetermines whether or not the moving average exceeds a predeterminedthreshold. Even with a very reddish intraesophageal image or the likepossibly resulting from a variation in illumination conditions caused bya variation in observation distance, angle, or the like, the use of sucha moving average makes it possible to eliminate the adverse effect of aslight variation in average brightness value to more accuratelydetermine that, for example, the capsule endoscope 103 has passedthrough the EG junction.

Further, in the above example, the luminance calculated from the threepixel values for R, G, and B as described above is used as a brightnessfeature value. As a fourth variation, G or B pixel data may be used inplace of the luminance.

FIG. 41 is a schematic graph illustrating a variation in G or B pixeldata on a series of endoscopic images in the case in which G or B pixeldata is used in place of the luminance calculated from the three pixelvalues for R, G, and B as described above. In FIG. 41, the axis ofabscissa indicates the image numbers (frame numbers), along the timeseries, of endoscopic images of the area extending from the esophagussquamous epithelium through the EG junction to the stomach. The axis ofordinate indicates the G or B pixel data value for the endoscopic imagecorresponding to each of the image numbers.

That is, as shown in FIG. 41, the G or B pixel data value varies betweenthe squamous epithelium RA of the esophagus and the stomach portion RCof the columnar epithelium. Further, the G or B pixel data value for theEG junction RB, located between the squamous epithelium RA and thestomach portion RC, is different from those for the squamous epitheliumRA and the stomach portion RC. Specifically, in the EG junction RB, theG or B pixel data value decreases gradually from the squamous epitheliumRA of the esophagus, which has a white color tone and thus a greaterpixel data value, to the stomach portion RC of the columnar epithelium,which has a smaller pixel data value.

Accordingly, when the brightness information on the image, varyinggradually, exhibits a value smaller than a predetermined threshold Th12, the apparatus determines that, for example, the capsule endoscope103 is passing through the EG junction or is about to enter the EGjunction. That is, on the basis of a variation in the brightnessinformation on the picked-up image, the apparatus determines that, forexample, the capsule endoscope 103 is passing through the EG junction.Specifically, to make a reliable determination for the passage or thelike, the average value of the G or B pixel data value is utilized asbrightness information.

Moreover, as a fifth variation, a plural pieces of brightnessinformation may be used. For example, in step S233 in FIG. 39, theluminance value and G pixel data for the pixel calculated from the threepixel values for R, G, and B are used to calculate the average values ofthese values, that is, the average brightness value I1 i of all thepixels in each image and the average value I2 i of the G pixel data. Instep S234, the apparatus determines whether or not for the averagebrightness value I1 i and the average value I2 i for the G pixel data,I1 i>Th13 and I2 i<Th14.

Further, as a sixth variation, the passage of the capsule endoscope 103through the EG junction or the like may be detected on the basis of theamount of variation in brightness information. That is, the apparatusmay determine, instead of whether or not the brightness informationobtained from each image of the series of consecutive images exceeds apredetermined threshold, whether or nor the amount of variation in thebrightness information on two consecutive images exceeds a predeterminedthreshold. That is, for the brightness information on each image, theaverage luminance value for the image is compared with that for thepreceding or succeeding image. If the difference between the two averageluminance values exceeds a predetermined threshold, the apparatus maydetermine that, for example, the capsule endoscope 103 has moved fromthe esophagus into the EG junction or from the EG junction into thestomach. The apparatus determines whether or not the differential value(Ii−I(i−m1)) between the average luminance values I(i−m1) and Ii for theimages F(i−m1) and Fi has varied by a predetermined threshold or more.m1 is 1, 2, 3, . . . .

The color tone of the mucosa may vary owing to individual differences inmucosa color, the presence of a lesion such as the Barrett esophagus, ora variation among image pickup systems. The sixth variation thus makesit possible to determine whether or not, for example, the capsuleendoscope 3 has passed through the EG junction without undergoing theadverse effect of the individual differences or the like.

Moreover, in this case, a variation in brightness information may bedetected by calculating the differential value of the average luminancevalues.

FIG. 42 is a flowchart showing an example of the flow of a process ofdetecting a variation in brightness by calculating the differentialvalue of the average luminance values. Steps similar to those shown inFIG. 35 are denoted by the same step numbers and the description ofthese steps is simplified.

As described above in conjunction with the process shown in FIG. 39, theimage data on each image is subjected to a preprocess such as inversegamma correction or noise removal before the process shown in FIG. 42 isexecuted. The process from step S201 to step S233 is the same as thatfrom step S201 to step S233 shown in FIG. 39.

The apparatus determines whether or not the process has been executed onall the images, that is, the process has been finished on all the images(step S221). If the process has not been finished on all the images, thedetermination in step S221 is NO. Then, a process of changing i to i+1is executed (step S207), and the process shifts to step S202.

When the process has been finished on all the images, the determinationin step 221 is YES, and a moving average value f(Ii) is calculated overa predetermined range, that is, over a predetermined number ofconsecutive images for smoothing (step S252). A differential valueΔf(Ii) is calculated on the basis of a temporal variation in the movingaverage value f(Ii) (step S253).

The image Fi is identified and detected which corresponds to thedifferential value Δf(Ii) exceeding a predetermined threshold thf1 (stepS254). Steps S233 to S254 constitute a detection section that detectsthe boundary of the gastrointestinal tract.

This enables the detection of a plurality of images for which the amountof variation in brightness exceeds the threshold. Even with individualdifferences in mucosa color or the like, the apparatus can determinewhether or not, for example, the capsule endoscope 103 has passedthrough the EG junction without undergoing the adverse effects of theindividual differences or the like.

Moreover, as a seventh variation, the closed cardia may be detected onthe basis of the distribution of brightness. For example, instead of theaverage value for brightness information, the standard deviation or thevariance may be used as is the case with the fourth embodiment. Thismakes it possible to determine whether or not, for example, the capsuleendoscope 103 is passing through the EG junction on the basis of thestandard deviation or variance of brightness information on a series ofendoscopic images obtained. Specifically, the standard deviation of thebrightness of R image data is determined. If the standard deviation issmaller than a predetermined threshold, the apparatus determines thatthe closed cardia is being viewed from the front. This is because thebrightness of the image is relatively uniform when the closed cardia isviewed from the front. Moreover, instead of the standard deviation orvariance of the average luminance value, a variation coefficient for thestandard deviation or variance (=standard deviation/average brightnessvalue) may be used.

Moreover, the above examples use the pixel data on all the pixels ineach frame image. However, as an eighth variation, instead of processingall the pixels, only the pixels in predetermined regions of each framemay be sampled for processing as described above with reference to FIG.37 for the fourth embodiment. FIG. 37 shows an example of regions ineach frame image which are to be subjected to the image processing inaccordance with the above present embodiment and variations.

Each frame image is divided into predetermined regions. In FIG. 37, eachimage is divided into 16 rectangular regions. The above process isexecuted only on preset ones (R2, R3, R5, R8, R9, R12, R14, and R15) ofthe resulting regions, that is, only on the regions of interest (ROI).In particular, since the esophagus is a luminal organ, the regions otherthan the one corresponding to the center of the visual field may be setto be the regions of interest (ROI) in order to more accuratelycalculate the color tone of the mucosa surface.

Accordingly, processing only the regions of interest (ROI) reduces theamount of calculation required, enabling an increase in processingspeed.

Moreover, when only the regions of interest (ROI) are processed, theprocessing speed may further be increased by, instead of processing allthe frames, processing only the pixels in the regions of interest (ROI)in every k (k=1, 2, 3, . . . ) frames. In particular, a large number ofimages are picked up for the interior of the esophagus. Consequently,accurate determinations may be made in spite of minor decimations.

As described above, the present embodiment (including the variations)makes it possible to determine, on the basis of brightness informationon each image, whether or not, for example, the image shows that thecapsule endoscope is about to enter the EG junction or is passingthrough the EG junction.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect whether or not eachimage shows that the capsule endoscope is about to enter the EG junctionor is passing through the EG junction. However, for example, anidentification function such as a well-known linear discriminationfunction may be used for the detection. Alternatively, a feature valuein accordance with another embodiment may be combined with the presentembodiment.

Sixth Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with a sixth embodiment.Endoscopic images to be processed in accordance with the presentembodiment are a series of endoscopic images picked up by the capsuleendoscope apparatus 101 as in the case of the fourth embodiment.Accordingly, the configuration of the cardia detection apparatus issimilar to that in the fourth embodiment and will not be describedbelow.

The above fourth embodiment uses the color tone feature value. However,the cardia detection apparatus, a luminal image processing apparatus inaccordance with the present embodiment, is characterized by detectingthe open cardia to determine whether or not each image shows that, forexample, the capsule endoscope moving from the esophagus toward thestomach is located near and in front of the cardia.

FIG. 43 is a diagram showing an example of an image in which the capsuleendoscope 103 is located in front of the open cardia.

When the capsule endoscope 103 picks up an image of the open cardia inthe lumen, the brightness of the open cardia is significantly lower thanthat of the surroundings. As shown in FIG. 43, when the cardia 132 isopen, in the image 131 picked up by the capsule endoscope 103, the opencardia 132 appears to be a dark area. Accordingly, as the capsuleendoscope 103 moves from the esophagus, through the EG junction, andcloser to the cardia, the area of the cardia 132 in the image 131increases. According to the present embodiment, when the area of theopen cardia exceeds a predetermined size, the apparatus determines thatthe capsule endoscope 103 moving from the stomach side of the esophagusis passing through the cardia.

FIG. 44 is a flowchart showing an example of the flow of a process ofdetecting the open cardia on the basis of a series of endoscopic imagesobtained. The series of endoscopic images picked up by the endoscopeswallowed through the subject's mouth comprise a plurality of frames.The process shown in FIG. 44 is executed on each of the frames. Theimage data on each endoscopic image is subjected to a preprocess such asinverse gamma correction or noise removal before the process shown inFIG. 44 is executed.

To start processing with the first frame of the series of images to besubjected to the process shown in FIG. 44, first, the frame number i isset at 1 (step S261). Reference character i denotes an integer from 1 ton.

Then, R image data on the image Fi with the frame number i is read fromthe storage device (not shown) in the terminal apparatus 7 (step S262).The image Fi comprises three planes for R, G, and B. In this case, onlythe R image data is read.

Although in this case, the R image data is read in order to make acardia determination as described below, G image data or B image data onthe image Fi may be used.

A dark portion pixel binarization is executed on all the pixels in theread R image data on the image Fi (step S263). Specifically, the pixelvalue of each pixel is compared with the predetermined threshold Th2 toexecute a binarization such that dark portion pixels are set to have avalue of 1, whereas the other pixels are set to have a value of 0(zero). Reference character j denotes a number identifying a pixel inthe image data on each frame. Then, whether or not the value rj for eachpixel is smaller than the threshold Th2 is checked to set 1 for pixelshaving a value smaller than the threshold Th2, while setting 0 (zero)for the other pixels.

Then, the ratio ε of the dark pixels to all the pixels is calculated(step S264). In other words, the ratio ε is the rate of the area of thedark portion in the image. Specifically, the ratio ε is calculated bydividing the number of pixels determined to be dark portion pixels bythe binarization in step S263, by the total number of the pixels in theR image data. When the number of dark portion pixels is defined as p1and the size of the image Fi is defined as ISX×ISY, the ratio ε isp1/(ISX'ISY). Step S264 constitutes a dark portion ratio calculationstep or a dark portion ration calculation section which calculates theratio of the dark portion pixels in each image Fi.

The ratio ε is compared with the predetermined threshold Thr todetermine whether or not the ratio ε of the dark portion pixels exceedsthe predetermined threshold Thr (step S265). The threshold Thr is, forexample, 0.8.

If the determination in step S265 is YES, that is, the dark portionaccounts for more than 80% of the entire image, the image is determinedto show the open cardia (step S266). The process is then ended.

If the determination in step S265 is NO, that is, the dark portionaccounts for at most 80% of the entire image, the image is determinednot to show the cardia (step S267). The apparatus determines whether ornot the process shown in FIG. 44 has been finished on all of the seriesof images to be subjected to the process shown in FIG. 44 (step S268).When the process has been finished on all the images, the determinationin step S268 is YES and the process is ended. If the determination instep S268 is NO, there remains an unprocessed image. Thus, a process ofchanging to i+1 is executed (step S269). Subsequently, steps S262 toS265 are repeatedly executed on the next image.

In step S266, when the ratio c of the dark portion pixels exceeds thepredetermined threshold Thr, the apparatus determines that the opencardia has been detected. In other words, the apparatus may alsodetermine that the endoscope will subsequently pass through the cardiaor reach the interior of the stomach. Steps S263 to S266 constitute adetection section that detects the boundary of the gastrointestinaltract.

Now, variations of the present embodiment will be described below.

In the above example, only the R image data is read in order to make acardia determination. However, G or B image data or G and B image datamay further be read, and a dark portion pixel binarization may beexecuted on the at least two image data. Then, when the ratios ε of thedark portion pixels in the at least two image data all exceed thepredetermined threshold Thr, the apparatus may determine that the cardiahas been detected.

In FIG. 44, described above, the cardia is detected on the basis of eachframe image. However, as a second variation, the apparatus may determinethat the cardia has been detected when the determination in step S265 isε>Thr for a plurality of consecutive images or at least a predeterminedrate (for example, 80%) of the plurality of consecutive images.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a third variation, theprocess shown in FIG. 44 may be executed on one image.

Moreover, as a fourth variation, a cardia determination may be made by,instead of calculating the ratio ε of the dark portion pixels to all thepixels, for example, calculating and dividing the total number of pixelsin the non-dark portion, in the above example, pixels P0 having a valueof at least the threshold Th2, by the total number of pixels in theimage to calculate the ratio cl of the pixels in the non-dark portion,or calculating the ratio of the number p0 of the non-dark portion pixelsto the number p1 of the dark portion pixels (p0/p1; p1 is not 0 (zero)).

Further, as a fifth variation, the threshold Thr may be varied dependingon the distance between the capsule endoscope 103 and the cardiaobserved when the cardia is detected. In other words, the threshold Thrmay be varied depending on the distance between the capsule endoscope 3and the cardia observed when the cardia is to be detected. For example,setting Thr at 0.5 enables the cardia to be detected more quickly thansetting Thr at 0.8 as described above.

As described above, the present embodiment enables the cardia to bedetected on the basis of the area of the dark portion in the image.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect the open cardia.However, for example, an identification function such as a well-knownlinear discrimination function may be used for the detection.Alternatively, a feature value in accordance with another embodiment maybe combined with the present embodiment.

Seventh Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with a seventh embodiment.The present embodiment is characterized by detecting the open cardia onthe basis of a shape. Endoscopic images to be processed in accordancewith the present embodiment are a series of endoscopic images picked upby the capsule endoscope apparatus 101 as in the case of the fourthembodiment. Accordingly, the configuration of the cardia detectionapparatus is similar to that in the fourth embodiment and will not bedescribed below.

The cardia detection apparatus, a luminal image processing apparatus inaccordance with the present embodiment, is characterized by detectingthe shape of the open cardia to determine whether or not each imageshows that, for example, the capsule endoscope moving from the esophagustoward the stomach is located near and in front of the cardia.

FIG. 45 is a diagram showing an example of an image picked up when thecapsule endoscope 103 passes through the open cardia. An image 131Acontains the open cardia 132A, corresponding to a dark image.

FIG. 46 is a flowchart showing an example of the flow in which the opencardia is detected on the basis of a series of endoscopic imagesobtained. The series of endoscopic images picked up by the endoscopeswallowed through the subject's mouth comprise a plurality of frames.The process shown in FIG. 46 is executed on each of the frames. Theimage data on each endoscopic image is subjected to a preprocess such asinverse gamma correction or noise removal before the process shown inFIG. 46 is executed.

To start processing with the first frame of the series of images to besubjected to the process shown in FIG. 46, first, the frame number i isset at 1 (step S271). Reference character i denotes an integer from 1 ton.

Then, R image data on the image Fi with the frame number i is read fromthe storage device (not shown) in the terminal apparatus 7 (step S272).The image Fi comprises three planes for R, G, and B. In this case, onlythe R image data is read.

A bandpass filtering process is executed on the read R image data on theimage Fi (step S273). The bandpass filtering is implemented by aconvolution process using a well-known digital filter or on a Fouriersurface.

The bandpass filter has, for example, such a property as shown in FIG.47. FIG. 47 is a diagram showing the filter property of a bandpassfiltering process. The filter property is such that a large number ofpassing components are present in a slightly low frequency band in orderto suppress the adverse effect of fine edge components such as vesselsas shown in FIG. 47. The filter property is such that, for example,within a spatial frequency band from 0 (zero) to π (rad), the passingproperty exhibits a peak value of (1.0) at π/4.

Then, a binarization is executed on edge components of an imageresulting from the bandpass filtering process, using a threshold (stepS274). The binarization sets a predetermined threshold Th3 at 10.0 andextracts an edge component exhibiting a variation of a value larger thanthe threshold Th3. Pixels with the edge component exhibiting a variationof a value larger than the threshold Th3 are set to have a value 1,whereas the other pixels are set to have a value of 1. The other pixelsare set have a value of 0 (zero). Reference character j denotes a numberidentifying a pixel in the image data on each frame. Then, whether ornot the value for the edge component of each pixel exceeds the thresholdTh3 is checked to set 1 for the pixels rj having a value larger than thethreshold Th3, while setting 0 (zero) for the other pixels. As describedabove, an image containing the cardia exhibits a rapid variation inbrightness. Accordingly, setting a higher threshold Th3 enables theexclusion of other edge components, for example, wrinkles resulting fromthe deformation of the mucosa.

Then, a cardia determination process is executed to determine whether ornot the image with the extracted edge component results from the cardiato determine whether or not the image shows the cardia (step S275). Thepresent embodiment determines whether or not the image shows the cardiaby thinning the extracted edge component and using a coincidence with anapproximate circle as an evaluation value to determine whether or notthe edge component is shaped generally like a circle. Whether or not theedge component is shaped generally like a circle is determined by a Haffconversion or the like.

FIG. 48 is a diagram showing an example of an image showing the resultof execution of a predetermined process of bandpass filtering andbinarization on the image shown in FIG. 45. As shown in FIG. 48, theopen cardia has a generally circular shape 132B. Accordingly, the Haffconversion or the like is executed on the edge component image 131B inFIG. 48 to determine whether or not the edge component 132B is shapedlike a circle.

If the image is determined to show the cardia on the basis of the resultof the cardia determination process, the determination in step S275 isYES. The apparatus thus determines that the cardia has been detected(step S276). The process is then ended.

If the determination in step S275 is NO, that is, the image does notshow the cardia, then the apparatus determines that the target site isnot the cardia (step S277). The apparatus then determines whether or notthe process shown in FIG. 46 has been finished on all of the series ofimages to be subjected to the process shown in FIG. 46 (step S278). Whenthe process has been finished on all the images, the determination instep S278 is YES. The process is then ended. If the determination instep S278 is NO, there remains an unprocessed image. Thus, a process ofchanging i to i+1 is executed (step S279). Subsequently, steps S272 toS274 are repeatedly executed on the next image. Steps S273 to S276constitute a feature value calculation section and a detection sectionthat detects the boundary of the gastrointestinal tract.

Now, variations of the present embodiment will be described below.

In the above example, only the R image data is read in order to make acardia determination. However, G or B image data or G and B image datamay further be read so that a dark portion pixel binarization can beexecuted on the at least two image data. Then, when the dark portionpixel is determined to have a circular shape in the at least two imagedata, the apparatus may determine that the cardia has been detected.

In FIG. 46, described above, the cardia is detected on the basis of eachframe image. However, as a second variation, the apparatus may determinethat the cardia has been detected when the dark portion pixel isdetermined to have a circular shape in a plurality of consecutive imagesor at least a predetermined rate (for example, 80%) of the plurality ofconsecutive images.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a third variation, theprocess shown in FIG. 46 may be executed on one image.

As described above, the present embodiment enables the cardia to bedetected on the basis of the shape of the open cardia in the image.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect the open cardia.However, for example, an identification function such as a well-knownlinear discrimination function may be used for the detection.Alternatively, a feature value in accordance with another embodiment maybe combined with the present embodiment.

Eighth Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with an eighth embodiment.The present embodiment is characterized by, in detecting the open cardiaon the basis of the shape, utilizing both the detection of the darkportion region boundary and the edge detection, described in the seventhembodiment, to determine whether or not a picked-up image contains theopen cardia. Endoscopic images to be processed in accordance with thepresent embodiment are a series of endoscopic images picked up by thecapsule endoscope apparatus 1 as in the case of the fourth embodiment.Accordingly, the configuration of the cardia detection apparatus, aluminal image processing apparatus, is similar to that in the fourthembodiment and will not be described below.

FIG. 49 is a flowchart showing an example of the flow in which thecardia is detected on the basis of a series of endoscopic imagesobtained. The series of endoscopic images picked up by the endoscopeswallowed through the subject's mouth comprise a plurality of frames.The process shown in FIG. 49 is executed on each of the frames. Theimage data on each endoscopic image is subjected to a preprocess such asinverse gamma correction or noise removal before the process shown inFIG. 49 is executed.

To start processing with the first frame of the series of images to besubjected to the process shown in FIG. 49, first, the frame number i isset at 1 (step S281). Reference character i denotes an integer from 1 ton.

Then, R image data on the image Fi with the frame number i is read fromthe storage device (not shown) in the terminal apparatus 7 (step S282).The image Fi comprises three planes for R, G, and B. In this case, onlythe R image data is read.

A dark portion pixel binarization is executed on all the pixels in theread R image data on the image Fi (step S283). Specifically, the pixelvalue of each pixel is compared with the predetermined threshold Th2 toexecute a binarization such that dark portion pixels are set to have avalue of 1, whereas the other pixels are set to have a value of 0(zero). Reference character j denotes a number identifying a pixel inthe image data on each frame. Then, whether or not the value rj for eachpixel is smaller than the threshold Th2 is checked to set 1 for pixelshaving a value smaller than the threshold Th2, while setting 0 (zero)for the other pixels.

Then, a boundary extraction process is executed to extract the boundaryof the extracted dark portion, in other words, to extract thecorresponding edge (step S284). The boundary extraction processinvolves, for example, using a pixel with a value of 1 (dark portionpixel) as a pixel of interest to set a mask region of size 3×3 aroundthe pixel of interest, and if any of the eight pixels in the mask regionhas a value of 0 (zero), defining the pixel of interest as a boundarypixel having a value of 1. Step S284 constitutes a dark portion boundaryextraction step or a dark portion boundary extraction section whichextracts the boundary of the dark portion pixel from each image Fi. Theboundary extraction process is executed on all the dark portion pixels.FIG. 50 shows an image of an extracted boundary. Executing boundaryextraction on an image 131C shown in FIG. 50 results in a generallycircular shape 132C along the boundary of the open cardia.

Then, a bandpass filtering process is executed on the read R image dataon the image Fi (step S285). The bandpass filtering is implemented by aconvolution process using a well-known digital filter or on a Fouriersurface as described in the seventh embodiment.

The bandpass filter has, for example, such a property as shown in FIG.47, described above. The filter property is such that a large number ofpassing components are present in a slightly low frequency band in orderto suppress the adverse effect of fine edge components such as vessels.

Then, a binarization is executed on edge components of an imageresulting from the bandpass filtering process, using a threshold (stepS286). As described in the seventh embodiment, the binarization, forexample, extracts an edge component exhibiting a variation of a valuelarger than the threshold Th3. Pixels with the edge component exhibitinga variation of a value larger than the threshold Th3 are set to have avalue 1, whereas the other pixels are set to have a value of 0 (zero).Reference character j denotes a number identifying a pixel in the imagedata on each frame. Then, whether or not the value for the edgecomponent of each pixel exceeds the threshold Th3 is checked to set 1for the pixels rj having a value larger than the threshold Th3, whilesetting 0 (zero) for the other pixels. As described above, an imagecontaining the cardia exhibits a rapid variation in brightness.Accordingly, setting a higher threshold Th3 enables the exclusion ofother edge components, for example, wrinkles resulting from thedeformation of the mucosa. An expansion process may be executed on theextracted pixel in order to suppress the adverse effect of theinaccurate detection of the dark portion boundary.

Then, a calculation is made of the coincidence between the boundarypixel extracted in step S284 and the edge component extracted pixelextracted in step S286 as an edge component (step S287). Specifically,the apparatus determines, for each extracted boundary pixel ek1 (k1=1,2, 3, . . . , K; reference character K denotes the total number ofpixels detected as boundary pixels), whether or not a pixel with thesame coordinates on the image has also been extracted as an edgecomponent pixel. The number n1 of pixels which are boundary pixels ek1and which have also been extracted as edge component pixels is counted.This is achieved by performing the logical AND of the boundary pixel ek1and the edge component of the pixel with the same coordinates on theimage. The ratio (n1/K) of the number n1 to the total number K ofboundary pixels is calculated.

Then, the apparatus determines whether or not the image contains thecardia on the basis of the ratio (n1/K) indicating the coincidencebetween the boundary pixel and the edge component extracted pixel (stepS288). FIG. 51 is a diagram showing an example of an image obtained byexecuting a predetermined process of bandpass filtering and binarizationon the processing target image. The open cardia has a generally circularshape 32B. However, FIG. 51 shows a break in a part of the circle. Inthis case, the ratio of the boundary of the dark portion region in FIG.50 to the edge component in FIG. 51 is calculated to determine whetheror not the image shows the cardia.

If the image is determined to show the cardia on the basis of the resultof the cardia determination process, the determination in step S288 isYES. The apparatus thus determines that the cardia has been detected(step S289). The process is then ended. Specifically, when the ratio(n1/K) exceeds a predetermined threshold the, the determination in stepS288 is YES. The apparatus thus determines that the cardia has beendetected (step S289). The process is then ended.

If the determination in step S288 is NO, that is, the ratio (n1/K) doesnot exceed the predetermined threshold the, then the apparatusdetermines that the target site is not the cardia (step S290). Theapparatus then determines whether or not the process shown in FIG. 49has been finished on all of the series of images to be subjected to theprocess shown in FIG. 49 (step S291). When the process has been finishedon all the images, the determination in step S291 is YES. The process isthen ended. If the determination in step S291 is NO, there remains anunprocessed image. Thus, a process of changing i to i+1 is executed(step S292). Subsequently, steps S282 to S288 are repeatedly executed onthe next image. Steps S283 to S289 constitute a feature valuecalculation section and a detection section that detects the boundary ofthe gastrointestinal tract.

Now, variations of the present embodiment will be described below.

In the above example, only the R image data is read in order to make acardia determination. However, G or B image data or G and B image datamay further be read so that a dark portion pixel binarization can beexecuted on the at least two image data. Then, when the coincidencebetween the at least two image data exceeds the predetermined threshold,the apparatus may determine that the cardia has been detected.

In FIG. 49, described above, the cardia is detected on the basis of eachframe image. However, as a second variation, the apparatus may determinethat the cardia has been detected when the coincidence exceeds thepredetermined threshold in a plurality of consecutive images or at leasta predetermined rate (for example, 80%) of the plurality of consecutiveimages.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a third variation, theprocess shown in FIG. 49 may be executed on one image.

As described above, the present embodiment determines whether or not thearea extracted as a dark portion has a large edge and can thus make anaccurate cardia determination.

Further, the present embodiment can make an accurate cardiadetermination even if the cardia is not circularly open or if the longdistance between the capsule endoscope 3 and the cardia results in adistant view of the cardia.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect the open cardia.However, for example, an identification function such as a well-knownlinear discrimination function may be used for the detection.Alternatively, a feature value in accordance with another embodiment maybe combined with the present embodiment.

Ninth Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with a ninth embodiment.The present embodiment is characterized by, in detecting the open cardiaon the basis of the shape, utilizing both the detection of the darkportion centroid and the edge detection, described in the seventhembodiment, to determine whether or not a picked-up image contains theopen cardia. Endoscopic images to be processed in accordance with thepresent embodiment are a series of endoscopic images picked up by thecapsule endoscope apparatus 101 as in the case of the fourth embodiment.Accordingly, the configuration of the cardia detection apparatus, aluminal image processing apparatus, is similar to that in the fourthembodiment and will not be described below.

Process steps similar to those in the eighth embodiment are denoted bysimilar step numbers and the description of these steps is simplified.

FIG. 52 is a flowchart showing an example of the flow in which the opencardia is detected on the basis of a series of endoscopic imagesobtained. In FIG. 52, process steps similar to those in the eighthembodiment are denoted by similar step numbers and the description ofthese steps is simplified.

The series of endoscopic images picked up by the endoscope swallowedthrough the subject's mouth comprise a plurality of frames. The processshown in FIG. 52 is executed on each of the frames. The image data oneach endoscopic image is subjected to a preprocess such as inverse gammacorrection or noise removal before the process shown in FIG. 52 isexecuted.

To start processing with the first frame of the series of images to besubjected to the process shown in FIG. 52, first, the frame number i isset at 1 (step S281). Reference character i denotes an integer from 1 ton.

Then, R image data on the image Fi with the frame number i is read fromthe storage device (not shown) in the terminal apparatus 7 (step S282).

A dark portion pixel binarization is executed on all the pixels in theread R image data on the image Fi (step S283). Specifically, the pixelvalue of each pixel is compared with the predetermined threshold Th2 toexecute a binarization such that dark portion pixels are set to have avalue of 1, whereas the other pixels are set to have a value of 0(zero). Reference character j denotes a number identifying a pixel inthe image data on each frame. Then, whether or not the value rj for eachpixel is smaller than the threshold Th2 is checked to set 1 for pixelshaving a value smaller than the threshold Th2, while setting 0 (zero)for the other pixels.

Then, on the basis of coordinate data on the dark portion regiondetected by the binarization, the coordinates of the centroid of thedark portion region are calculated (step S301). Step S301 constitutes adark portion region centroid coordinate calculation step or a darkportion region centroid coordinate calculation section which calculatesthe coordinates of the centroid of the dark portion region in the imageFi. FIG. 53 shows the centroid position calculated by the dark portionregion centroid coordinate calculation section. In an image 131B in FIG.53, the centroid 132Ec of the dark portion region 132E of the generallycircular open cardia is shown at the position of the calculated centroidcoordinates.

Then, a bandpass filtering process is executed on the read R image dataon the image Fi as described in the eighth embodiment (step S285).Moreover, an edge component binarization is executed on the imageresulting from the bandpass filtering process, using a threshold (stepS286).

In step S286, circumferential character is evaluated by determiningwhether or not the pixel of an edge component extracted in step S286lies so as to surround the dark portion region (step S307).Specifically, as shown in FIG. 54, the apparatus determines whether ornot the extracted edge component pixel is penetrated in predeterminedradial directions from the calculated centroid coordinates 132Ec. FIG.54 is intended to describe the evaluation of the circumferentialcharacter. In the example shown in FIG. 54, the number m2 of those of aplurality of radial lines 132Ed extending from the centroid coordinates132Ec which cross the edge component pixel (this number is hereinafterreferred to as the intersection count) is counted. Then, the lines 132Edcross or penetrate the edge component in seven of the eight directions.Consequently, the intersection count m2 is 7.

Then, the apparatus determines whether or not the image contains thecardia on the basis of the intersection count m2 (step 288). As shown inFIG. 54, even with a break in a part of the circle formed by the edgecomponent pixel, the apparatus determines the image to contain thecardia when the ratio (m2/m3) of the intersection count m2 to the numberm3 of the plurality of lines 132Ed exceeds a predetermined thresholdThm. In other words, whether or not the image shows the cardia isdetermined on the basis of the ratio (m2/m3) of those of the pluralityof lines extending radially from the centroid position of the darkportion region in FIG. 53 which cross the edge component pixel in FIG.54.

If the ratio of the intersection count m2 to the number m3 of theplurality of segments 132Ed exceeds the threshold Thm, the determinationin step S288 is YES. The apparatus thus determines that the cardia hasbeen detected (step S289). The process is then ended. Specifically, whenthe ratio (m2/m3), indicating an evaluative value for thecircumferential character, exceeds the predetermined threshold thm, forexample, 0.7, the determination in step S88 is YES. The apparatus thusdetermines that the cardia has been detected (step S289). The process isthen ended.

If the determination in step S288 is NO, that is, the ratio (m2/m3) doesnot exceed the predetermined threshold thm, then the apparatusdetermines that the target site is not the cardia (step S290). Theapparatus then determines whether or not the process shown in FIG. 52has been finished on all of the series of images to be subjected to theprocess shown in FIG. 52 (step S291). When the process has been finishedon all the images, the determination in step S291 is YES. The process isthen ended. If the determination in step S291 is NO, there remains anunprocessed image. Thus, a process of changing i to i+1 is executed(step S292). Subsequently, steps S282 to S288 in FIG. 52 are repeatedlyexecuted on the next image. Steps S283 to S289 constitute a featurevalue calculation section and a detection section that detects theboundary of the gastrointestinal tract.

Now, variations of the present embodiment will be described below.

In the above example, only the R image data is read in order to make acardia determination. However, G or B image data or G and B image datamay further be read so that a dark portion pixel binarization can beexecuted on the at least two image data. Then, when the evaluative valuefor the circumferential character exceeds the predetermined thresholdfor the at least two image data, the apparatus may determine that thecardia has been detected.

In FIG. 52, described above, the cardia is detected on the basis of eachframe image. However, as a second variation, the apparatus may determinethat the cardia has been detected when the evaluative value for thecircumferential character exceeds the predetermined threshold for aplurality of consecutive images or at least a predetermined rate (forexample, 80%) of the plurality of consecutive images.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a third variation, theprocess shown in FIG. 52 may be executed on one image.

As a fourth variation, the circumferential character may be evaluated onthe basis of the number or ratio of those of a plurality ofpredetermined regions in which the edge component image is present. FIG.55 is a diagram illustrating that the evaluation of the circumferentialcharacter in accordance with the fourth variation is based on the ratioof regions. For example, as shown in FIG. 55, the image is divided into,for example, eight regions around the centroid 132Ec, the regionsspanning between 0° and 45°, between 45° and 90°, between 90° and 135°,between 135° and 180°, between 180° and 225°, between 225° and 270°,between 270° and 315°, and between 315° and 360°. The apparatus thendetermines whether or not each of the regions contains the edgecomponent pixel obtained in step S286. When the number of regionscontaining the edge component pixel 132G is at least the predeterminedthreshold, the apparatus may determine that the cardia has beendetected. FIG. 56 shows that the edge component pixel is present in 6shaded regions. If the threshold is 6, the apparatus determines, in thecase of FIG. 55, that the cardia has been detected.

Moreover, the apparatus may determine that the cardia has been detectedon the basis of the angular range in which the edge component pixel ispresent, instead of the number of regions. For example, as shown in FIG.56, the angular range θ around the centroid 132Ec in which the edgecomponent pixel is present is determined. The apparatus may determinethat the cardia has been detected on the basis of whether or not theangular range θ is equal to at least the predetermined threshold, forexample, at least 270°. FIG. 56 is a diagram illustrating that thecircumferential character is evaluated on the basis of the angularrange. If a plurality of edge component pixel lines are present, theapparatus may determine whether or not the cardia has been detected bycomparing the edge component pixel in the greatest angular range withthe predetermined threshold.

As described above, although the dark portion region binarization mayundergo a variation in the position of the boundary depending on thethreshold Th2, the present embodiment can make an accurate cardiadetermination without suffering from the adverse effect of a possiblevariation in boundary position.

Further, the present embodiment can make an accurate cardiadetermination even if the cardia is not circularly open or if the longdistance between the capsule endoscope 103 and the cardia results in adistant view of the cardia.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect the open cardia.However, for example, an identification function such as a well-knownlinear discrimination function may be used for the detection.Alternatively, a feature value in accordance with another embodiment maybe combined with the present embodiment.

Tenth Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with a tenth embodiment.Endoscopic images to be processed in accordance with the presentembodiment are a series of endoscopic images picked up by the capsuleendoscope apparatus 101 as in the case of the fourth embodiment.Accordingly, the configuration of the cardia detection apparatus, aluminal image processing apparatus, is similar to that in the fourthembodiment and will not be described below.

In the above sixth to ninth embodiments, the open cardia is detected.However, the cardia may be closed. The cardia detection apparatus inaccordance with the present embodiment is characterized by detecting theclosed cardia to determine whether or not each image shows that, forexample, the capsule endoscope moving from the esophagus toward thestomach is located near and in front of the cardia.

FIG. 57 is a diagram showing an example of an image in which the capsuleendoscope 103 is located in front of the cardia.

When the capsule endoscope 103 picks up an image of the closed cardia inthe lumen, the picked-up image contains no definite dark portion regionowing to the closed cardia. Moreover, the brightness in the screen isrelatively uniform because the periphery of the cardia is viewed fromthe front at the terminal of the esophagus.

The present embodiment determines that the closed cardia has beendetected when the area of the closed cardia is smaller than apredetermined size.

FIG. 58 is a flowchart showing an example of the flow in which theclosed cardia is detected on the basis of a series of endoscopic imagesobtained. The series of endoscopic images picked up by the endoscopeswallowed through the subject's mouth comprise a plurality of frames.The process shown in FIG. 58 is executed on each of the frames. Theimage data on each endoscopic image is subjected to a preprocess such asinverse gamma correction or noise removal before the process shown inFIG. 58 is executed. The process shown in FIG. 58 includes process stepssimilar to those of the process shown in FIG. 44. Accordingly, processsteps similar to those shown in FIG. 44 are denoted by similar stepnumbers and the description of these steps is simplified.

To start processing with the first frame of the series of images to besubjected to the process shown in FIG. 58, first, the frame number i isset at 1 (step S261). Reference character i denotes an integer from 1 ton.

Then, R image data on the image Fi with the frame number i is read fromthe storage device (not shown) in the terminal apparatus 7 (step S262).The image Fi comprises three planes for R, G, and B. In this case, onlythe R image data is read.

Although in this case, the R image data is read in order to make acardia determination described below, G or B image data on the image Fimay be used.

A dark portion pixel binarization is executed on all the pixels in theread R image data on the image Fi (step S263). Specifically, the pixelvalue of each pixel is compared with the predetermined threshold Th2 toexecute a binarization such that dark portion pixels are set to have avalue of 1, whereas the other pixels are set to have a value of 0(zero). Reference character j denotes a number identifying a pixel inthe image data on each frame. Then, whether or not the value rj for eachpixel is smaller than the threshold Th2 is checked to set 1 for pixelshaving a value smaller than the threshold Th2, while setting 0 (zero)for the other pixels.

Then, a labeling process is executed to label a plurality of regions ofthe dark portion pixel, and a dark portion region with the largest areaD is then identified (step S314). Step S314 constitutes a largest darkportion region identification step or a largest dark portion regionidentification section which identifies the dark portion region with thelargest area D in each image Fi. FIG. 57 shows an image 132I containinga dark portion region 132I with the largest area D.

The apparatus then determines whether or not the largest area D issmaller than a predetermined threshold Thd1 (step S315). The thresholdThd1 is, for example, 0.1.

If the determination in step S315 is YES, that is, the dark portionaccounts for less than 10% of the entire image, the apparatus determinesthat the cardia has been detected (step 266). The process is then ended.

If the determination in step S315 is NO, that is, the dark portion doesnot account for less than 10% of the entire image, the apparatusdetermines that the image does not show the cardia (step 267). Theapparatus then determines whether or not the process shown in FIG. 58has been finished on all of the series of images to be subjected to theprocess shown in FIG. 58 (step S268). When the process has been finishedon all the images, the determination in step S268 is YES. The process isthen ended. If the determination in step S268 is NO, there remains anunprocessed image. Thus, a process of changing i to i+1 is executed(step S269). Subsequently, steps S262 to S264 are repeatedly executed onthe next image.

In step S266, the apparatus determines that the cardia has been detectedwhen the area D of the largest dark portion region is smaller than thepredetermined threshold Thd1. In other words, the apparatus may alsodetermine that the capsule endoscope will subsequently pass through thecardia or reach the interior of the stomach. Steps S263 to S266constitute a feature value calculation section and a detection sectionthat detects the boundary of the gastrointestinal tract.

Now, variations of the present embodiment will be described below.

In the above example, only the R image data is read in order to make acardia determination. However, G or B image data or G and B image datamay further be read so that a dark portion pixel binarization can beexecuted on the at least two image data. Then, when the areas D of thedark portion regions with the largest area in the at least two imagedata are all smaller than the predetermined threshold Thd I, theapparatus may determine that the cardia has been detected.

In FIG. 58, described above, the cardia is detected on the basis of eachframe image. However, as a second variation, the apparatus may determinethat the cardia has been detected when the determination in step S315 isD<Thd1 for a plurality of consecutive images or at least a predeterminedrate (for example, 80%) of the plurality of consecutive images.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a third variation, theprocess shown in FIG. 58 may be executed on one image.

As a fourth variation, a region of interest (ROI) of such apredetermined size as shown by a dotted line in FIG. 57 may be setaround the above dark portion region with the largest area. The standarddeviation, variance, or variation coefficient of the brightness in theregion of interest (ROI) may then be calculated so that the apparatuscan determine that the closed cardia has been detected on the basis ofwhether or not the value of the standard deviation or the like issmaller than a predetermined threshold.

As described above, the present embodiment allows the closed cardia tobe detected on the basis of the area of the dark portion in the image.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect the closed cardia.However, for example, an identification function such as a well-knownlinear discrimination function may be used for the detection.Alternatively, a feature value in accordance with another embodiment maybe combined with the present embodiment.

Eleventh Embodiment

Now, with reference to the drawings, description will be given of acardia detection apparatus utilizing a capsule endoscope apparatus and amethod for the cardia detection in accordance with an eleventhembodiment. Endoscopic images to be processed in accordance with thepresent embodiment are a series of endoscopic images picked up by thecapsule endoscope apparatus 101 as in the case of the fourth embodiment.Accordingly, the configuration of the cardia detection apparatus issimilar to that in the fourth embodiment and will not be describedbelow.

The cardia detection apparatus, a luminal image processing apparatus inaccordance with the present embodiment, is characterized by detectingthe closed cardia to determine whether or not each image shows that, forexample, the capsule endoscope moving from the esophagus toward thestomach is located near and in front of the cardia.

FIG. 59 is a flowchart showing an example of the flow in which theclosed cardia is detected on the basis of a series of endoscopic imagesobtained. The series of endoscopic images picked up by the endoscopeswallowed through the subject's mouth comprise a plurality of frames.The process shown in FIG. 59 is executed on each of the frames. Theimage data on each endoscopic image is subjected to a preprocess such asinverse gamma correction or noise removal before the process shown inFIG. 59 is executed.

To start processing with the first frame of the series of images to besubjected to the process shown in FIG. 59, first, the frame number i isset at 1 (step S321). Reference character i denotes an integer from 1 ton.

Then, R image data on the image Fi with the frame number i is read fromthe storage device (not shown) in the terminal apparatus 7 (step S322).The image Fi comprises three planes for R, G, and B. In this case, onlythe R image data is read.

Although in this case, the R image data is read in order to make acardia determination described below, G or B image data on the image Fimay be used.

A dark portion pixel binarization is executed on all the pixels in theread R image data on the image Fi (step S323). Specifically, the pixelvalue of each pixel is compared with the predetermined threshold Th2 toexecute a binarization such that dark portion pixels are set to have avalue of 1, whereas the other pixels are set to have a value of 0(zero). Reference character j denotes a number identifying a pixel inthe image data on each frame. Then, whether or not the value rj for eachpixel is smaller than the threshold Th2 is checked to set 1 for pixelshaving a value smaller than the threshold Th2, while setting 0 (zero)for the other pixels.

Then, a thinning process is executed on the binarized image (step S324).FIG. 60 shows an image 131J containing a cardia shape 132J obtained bythinning the image of the closed cardia. FIG. 60 is a diagramillustrating the cardia shape obtained by thinning the image of theclosed cardia. Branching or intersecting points (hereinafter referred toas branching points) are calculated for each of the lines resulting fromthe thinning operation (step 325). Coordinate data on the branchingpoints is stored in the storage device as data indicating theconcentration level.

Then, the concentration level of the calculated branching points iscalculated (step S326). Step 326 constitutes a concentration levelcalculation step or a concentration level calculation section whichcalculates the concentration level of the branching points in each imageFi. For the concentration level, the variance of the coordinate valuesof the branching points is used as a parameter.

FIG. 61 is a flowchart showing an example of the flow of a process ofcalculating the variance value, which is a parameter for theconcentration level. First, coordinate data on the calculated branchingpoints is acquired from the storage device (step S341). For example,when N branching points have been calculated, the variance vx of the xcoordinate of each of the N branching points and the variance vy of they coordinate of the branching point are calculated (step S342). Thevariances vx, vy are stored for each of the N branching points.

Referring back to FIG. 59, the apparatus determines whether or not theconcentration level is high depending on whether or not the variancesvx, vy, which are data indicating the concentration level, are less thanthresholds thv1, thv2 (step S327).

If the determination in step S327 is YES, that is, the variances vx, vyare less than thresholds thv1, thv2, respectively, the apparatusdetermines that the cardia has been detected (step S328). The process isthen ended.

If the determination in step S327 is NO, that is, the variances vx, vyare not less than thresholds thv1, thv2, respectively, the apparatusdetermines that the image does not show the cardia (step S329). Theapparatus then determines whether or not the process shown in FIG. 59has been finished on all of the series of images to be subjected to theprocess shown in FIG. 59 (step S330). When the process has been finishedon all the images, the determination in step S330 is YES. The process isthen ended. If the determination in step S330 is NO, there remains anunprocessed image. Thus, a process of changing i to i+1 is executed(step S331). Subsequently, steps S322 to S327 are repeatedly executed onthe next image. Steps S323 to S328 constitute a feature valuecalculation section and a detection section that detects the boundary ofthe gastrointestinal tract.

FIG. 62 is a diagram showing an example of an image illustratingbranching points. In FIG. 62, for the closed cardia, branching points133J on a line in the image 131J concentrate in a part of the image131J. The level of the concentration is digitized on the basis of theabove variance to determine, for example, a variance value as describedabove. The level, for example, the variance value is compared with apredetermined threshold to determine whether or not the closed cardiahas been detected. The branching points concentrate in one part when theclosed cardia is thinned. Accordingly, the apparatus determines that thecardia has been detected when the branching points are found toconcentrate in one part.

In step S328, the apparatus determines that the closed cardia has beendetected when the variance value is smaller than the predeterminedthreshold. In other words, the apparatus may also determine that thecapsule endoscope 103 will subsequently pass through the cardia or reachthe interior of the stomach.

Now, variations of the present embodiment will be described below.

In the above example, only the R image data is read in order to make acardia determination. However, G or B image data or G and B image datamay further be read so that the concentration level can be calculatedfor the at least two image data. Then, the apparatus may determine thatthe cardia has been detected on the basis of the concentration levelsfor the at least two image data.

In FIG. 59, described above, the cardia is detected on the basis of eachframe image. However, as a second variation, the apparatus may determinethat the closed cardia has been detected when the determination in stepS327 is YES for a plurality of consecutive images or at least apredetermined rate (for example, 80%) of the plurality of consecutiveimages.

Further, in the above description, the process is executed on theplurality of consecutive images. However, as a third variation, theprocess shown in FIG. 29 may be executed on one image.

In accordance with the present embodiment, the threshold process isapplied to the calculated feature value to detect the closed cardia.However, for example, an identification function such as a well-knownlinear discrimination function may be used for the detection.Alternatively, a feature value in accordance with another embodiment maybe combined with the present embodiment.

The plurality of embodiments described above each make it possible todetect when, for example, the endoscope has passed through the EGjunction or is about to reach the cardia, on the basis of one endoscopicimage or a series of consecutive endoscopic images. This enables imagesrequired for diagnosing the esophageal disease to be selected from alarge number of picked-up endoscopic images, allowing the esophagealdisease to be quickly diagnosed.

Therefore, the plurality of embodiments described above can provide aluminal image processing apparatus that can detect the cardia.

For example, a determination for the Barrett esophagus is made on thebasis of an image of the vicinity of the EG junction. Thus, the abilityto detect the cardia as described above enables the disease to becarefully diagnosed while viewing only that image or only the imagepreceding or succeeding that image. This allows the diagnosis to bequickly achieved.

Description has been given of the example of the process using imagespicked up by the capsule endoscope 3. However, of course, the aboveprocess may be executed on images picked up by a normal endoscope, thatis, an endoscope having an elongate, flexible insertion portion. Boththe capsule endoscope and the normal endoscope can provide intraluminalimages.

Moreover, the cardia can be detected by each of the techniques inaccordance with the plurality of embodiments described above (includingthe variations). However, the cardia may be detected by a combination ofa plurality of the techniques.

The above fourth to eleventh embodiments and variations thereof canprovide a luminal image processing apparatus that can detect the cardiaon the basis of intraluminal images.

INDUSTRIAL APPLICABILITY

To make a determination for the Barrett esophagus condition or the likeon the basis of a large amount of endoscopic image data obtained bypicking up images of the interior of the esophagus, a process ofdetecting a first feature such as the EG junction or the epitheliumboundary which is located around the target site is repeated with aframe number sequentially changed until the presence of the feature isdetermined. For images succeeding the image determined to contain thefirst feature, the process shifts to detection of a second feature suchas the Barrett esophagus which corresponds to a determination target.This process is more efficient than a process of detecting the secondfeature and then making a determination therefor without detecting thefirst feature.

1. A medical image processing apparatus comprising: an image extractingsection that extracts a frame image from in vivo motion picture datapicked up by an in vivo image pickup device or a plurality ofconsecutively picked-up still image data; and an image analysis sectionthat analyzes the frame image extracted by the image extracting sectionto output an image analysis result, the image analysis sectioncomprising: a first biological feature detection section that detects afirst biological feature; a second biological feature detection sectionthat detects, on the basis of a detection result obtained by the firstbiological feature detection section, a second biological feature in aframe image picked up temporally before or after the image used fordetection by the first biological feature detection section; and acondition determination section that makes a determination for abiological condition on the basis of a detection result obtained by thesecond biological feature detection section to output the determination.2. The medical image processing apparatus according to claim 1, whereinon the basis of the detection result obtained by the first biologicalfeature detection section, the second biological feature detectionsection acquires a predetermined number of frame images to execute aprocess of detecting the second biological feature on the frame image.3. The medical image processing apparatus according to claim 1, whereinon the basis of the detection result obtained by the first biologicalfeature detection section, the second biological feature detectionsection sequentially acquires a frame image picked up temporally beforeor after the frame image processed by the first biological featuredetection section, and on the basis of the detection result obtained bythe first biological feature detection section by processing the frameimage acquired, the second biological feature detection section suspendsthe acquisition of the frame image.
 4. The medical image processingapparatus according to claim 1 , wherein the condition determinationsection makes a determination for the biological condition bydetermining whether or not the Barrett esophagus as an intraesophagealdisease is present.
 5. The medical image processing apparatus accordingto claim 4, wherein the first biological feature detected by the firstbiological feature detection section is an EG junction located in theesophagus as a boundary between the stomach and the esophagus.
 6. Themedical image processing apparatus according to claim 5, wherein thefirst biological feature detection section detects the EG junction byextracting coordinates of end points of palisade vessels and obtaining apalisade vessel end point boundary line formed of a segment connectingthe extracted end point coordinates.
 7. The medical image processingapparatus according to claim 6, wherein when a segment obtained byexecuting a thinning process on the frame image has at least apredetermined length, the first biological feature detection sectiondetermines the segment to be the palisade vessel.
 8. The medical imageprocessing apparatus according to claim 7, wherein the first biologicalfeature detection section determines the segment obtained by executingthe thinning process on the frame image data to be the palisade vessel,further taking into account the numbers of branching points,intersecting points, and bending points on the segment.
 9. The medicalimage processing apparatus according to claim 8, wherein the firstbiological feature detection section determines the segment obtained byexecuting the thinning process on the frame image data to be thepalisade vessel, further taking into account an angle between a segmentconnecting opposite ends of the segment and a vector connecting a darkportion center of an image dark portion of the frame image to one of theopposite ends which is closer to the image dark portion.
 10. The medicalimage processing apparatus according to claim 5, wherein the secondbiological feature detected by the second biological feature detectionsection is an epithelium boundary in the esophagus.
 11. The medicalimage processing apparatus according to claim 10, wherein the conditiondetermination section determines whether or not the Barrett esophagus ispresent on the basis of a distance between a plurality of intersectingpoints between the epithelium boundary and each of a plurality of radiallines radiating from a predetermined point and between the EG junctionand the radial line.
 12. The medical image processing apparatusaccording to claim 11, wherein the condition determination sectiondetermines whether or not the Barrett esophagus is present, taking intoaccount distances between a predetermined point and the epitheliumboundary crossing each of the plurality of radial lines radiating fromthe predetermined point or distances between the EG junction and thepredetermined point.
 13. The medical image processing apparatusaccording to claim 10, wherein the condition determination sectiondetermines whether or not the Barrett esophagus is present, further onthe basis of the number of points at which the palisade vessel crossesthe epithelium boundary.
 14. The medical image processing apparatusaccording to claim 13, wherein when a segment obtained by executing athinning process on the frame image has at least a predetermined length,the first biological feature detection section determines the segment tobe the palisade vessel.
 15. The medical image processing apparatusaccording to claim 14, wherein the first biological feature detectionsection determines the segment obtained by executing the thinningprocess on the frame image data to be the palisade vessel, furthertaking into account the numbers of branching points, intersectingpoints, and bending points on the segment.
 16. The medical imageprocessing apparatus according to claim 15, wherein the first biologicalfeature detection section determines the segment obtained by executingthe thinning process on the frame image data to be the palisade vessel,further taking into account an angle between a segment connectingopposite ends of the segment and a vector connecting a dark portioncenter of an image dark portion of the frame image to one of theopposite ends which is closer to the image dark portion.
 17. The medicalimage processing apparatus according to claim 1, wherein the firstbiological feature detected by the first biological feature detectionsection is the cardia serving as an inlet through which the apparatusmoves from the esophagus into the cardia, an inlet of the stomach, onthe basis of the detection result obtained by the first biologicalfeature detection section, the second biological feature detectionsection detects the second biological feature in the frame image pickedup temporally before the image used for detection by the firstbiological feature detection section.
 18. The medical image processingapparatus according to claim 17, wherein the first biological featuredetection section detects a thin line obtained by executing a thinningprocess on the frame image, as the closed cardia on the basis of aconcentration level of the calculated number of branching points orintersecting points.
 19. The medical image processing apparatusaccording to claim 18, wherein the first biological feature detectionsection detects the cardia as the open cardia on the basis of a thinnedimage generated by executing an edge extraction process and abinarization on the frame image and a dark portion image obtained byexecuting a dark portion extraction process of extracting an image darkportion on the frame image.
 20. The medical image processing apparatusaccording to claim 19, wherein the cardia is detected on the basis of aregion in which or an angle at which a thin line in the thinned image ispresent in a circumferential direction around a feature point in thedark portion.
 21. The medical image processing apparatus according toclaim 1, wherein the first biological feature detected by the firstbiological feature detection section is the epithelium boundarycorresponding to a boundary between the squamous epithelium in theesophagus and a columnar epithelium in the stomach.
 22. The medicalimage processing apparatus according to claim 21, wherein the conditiondetermination section makes a determination for the biological conditiononly for the frame image in which the epithelium boundary has beendetected, on the basis of the detection result obtained by the secondbiological feature detection section.
 23. A medical image processingmethod comprising: a step of extracting a frame image from in vivomotion picture data picked up by an in vivo image pickup device or aplurality of consecutively picked-up still image data; a step ofanalyzing the extracted frame image to detect a first biologicalfeature; a step of detecting, on the basis of a result of the detectionof the first biological feature, a second biological feature in a frameimage picked up temporally before or after the image used for detectionby the first biological feature detection section; and a step of makinga determination for a biological condition on the basis of a result ofthe detection of the second biological feature to output thedetermination.
 24. A program allowing a computer to execute: a functionof extracting a frame image from in vivo motion picture data picked upby an in vivo image pickup device or a plurality of consecutivelypicked-up still image data; a function of analyzing the extracted frameimage to detect a first biological feature; a function of detecting, onthe basis of a result of the detection of the first biological feature,a second biological feature in a frame image picked up temporally beforeor after the image used for detection by the first biological featuredetection section; and a function of making a determination for abiological condition on the basis of a result of the detection of thesecond biological feature to output the determination.
 25. A luminalimage processing apparatus comprising: a feature value calculatingsection that calculates a predetermined feature value by executing imageprocessing on one or a plurality of intraluminal images obtained bypicking up an image of the gastrointestinal tract; and a boundarydetection section that detects a boundary of the gastrointestinal tracton the basis of the calculated feature value.
 26. The luminal imageprocessing apparatus according to claim 25, further comprising adetermination section that determines that the intraluminal image showsa part extending from the esophagus to the cardia, on the basis of adetection result obtained by the boundary detection portion.
 27. Theluminal image processing apparatus according to claim 25, wherein thepredetermined feature value is a color tone of the one or plurality ofintraluminal images.
 28. The luminal image processing apparatusaccording to claim 25, wherein the predetermined feature value is abrightness of the one or plurality of intraluminal images.
 29. Theluminal image processing apparatus according to claim 25, wherein theboundary is the EG junction.
 30. The luminal image processing apparatusaccording to claim 25, wherein the feature value calculation sectioncalculates the feature value for pixels in the one or plurality ofintraluminal images other than dark portion pixels and halation pixels.31. The luminal image processing apparatus according to claim 25,wherein if the predetermined feature value is calculated from theplurality of intraluminal images, the determination section determinesthat the image shows the boundary when a differential value of movingaverage of the calculated predetermined feature values is at least apredetermined threshold.
 32. The luminal image processing apparatusaccording to claim 25, wherein the feature value calculation sectionexecutes image processing on pixels in a predetermined region in each ofthe one or plurality of intraluminal images to calculate thepredetermined feature value.
 33. The luminal image processing apparatusaccording to claim 25, wherein the predetermined feature value is anarea of a dark portion region or a non-dark portion region in the one orplurality of intraluminal images.
 34. The luminal image processingapparatus according to claim 33, wherein the boundary is the opencardia, and the boundary detection section detects the boundary bycomparing the area with a predetermined threshold.
 35. The luminal imageprocessing apparatus according to claim 25, wherein the predeterminedfeature value is a shape of the dark portion region in the one orplurality of intraluminal images.
 36. The luminal image processingapparatus according to claim 35, wherein the boundary detection sectiondetects the boundary depending on whether or not the shape is a circle.37. The luminal image processing apparatus according to claim 35,wherein the shape of the dark portion region includes a shape of aboundary of the dark portion region and an edge portion, and theboundary detection section detects the boundary by comparing acoincidence between the boundary and the edge portion with apredetermined threshold.
 38. The luminal image processing apparatusaccording to claim 35, wherein the shape of the dark portion regionincludes the edge portion of the dark portion region, and the boundarydetection section detects the boundary by comparing a rate at which theedge portion is present around a predetermined point in the dark portionregion, with a predetermined threshold.
 39. The luminal image processingapparatus according to claim 38, wherein the boundary detection sectiondetermines the rate on the basis of whether or not the edge portion ispresent in a plurality of predetermined areas in each of the one orplurality of intraluminal images.
 40. The luminal image processingapparatus according to claim 38, wherein the boundary detection sectiondetermines the rate on the basis of an angular range in which the edgeportion is present around a predetermined point in the dark portionregion.
 41. The luminal image processing apparatus according to claim25, wherein the predetermined feature value is an area of the darkportion region in the one or plurality of intraluminal images.
 42. Theluminal image processing apparatus according to claim 33, wherein theboundary is the closed cardia, and the boundary detection sectiondetects the boundary by comparing the area with a predeterminedthreshold.
 43. The luminal image processing apparatus according to claim42, wherein the boundary detection section detects the boundary byexecuting a labeling process to label a plurality of regions of a darkportion pixel in each of the one or plurality of intraluminal images anddetermining whether or not an area of one of the labeled plurality ofregions which has the largest area is smaller than the predeterminedthreshold.
 44. The luminal image processing apparatus according to claim25, wherein the boundary detection section detects the boundary byexecuting a thinning process on the dark portion pixel in each of theone or plurality of intraluminal images, calculating branching orintersecting points on each segment obtained by the thinning process,and executing the detection on the basis of a concentration level of thebranching or intersecting points in each of the images.
 45. The luminalimage processing apparatus according to claim 44, wherein the boundaryis the closed cardia, and the boundary detection section detects theboundary by comparing the concentration level with a predeterminedthreshold.
 46. A luminal image processing method comprising: a step ofcalculating a predetermined feature value by executing image processingon one or a plurality of intraluminal images obtained by picking up animage of the gastrointestinal tract; and a step of detecting a boundaryof the gastrointestinal tract on the basis of the calculated featurevalue.
 47. A program allowing a computer to execute: a function ofcalculating a predetermined feature value from one or a plurality ofintraluminal images obtained by picking up an image of thegastrointestinal tract; and a function of detecting a boundary of thegastrointestinal tract on the basis of the calculated feature value.